Soundtrack for electronic text

ABSTRACT

A soundtrack creation method and user playback system for soundtracks synchronized to electronic text. Synchronization is achieved by maintaining a reading speed variable indicative of the user&#39;s reading speed. The system provides for multiple channels of audio to enable concurrent playback of two or more partially or entirely overlapping audio regions so as to create an audio output having, for example, sound effects, ambience, music or other audio features that are triggered to playback at specific portions in the electronic text to enhance the reading experience.

FIELD OF THE INVENTION

The present invention relates to a method and system for making andplaying soundtracks for electronic text. In particular, the soundtracksare configured for playback that is synchronized with the reading speedof the user.

BACKGROUND TO THE INVENTION

Books are now commonly provided in various electronic text formats forreading on portable electronic devices with a display. For example, mostbooks now being published are provided as e-books, either exclusively orin parallel with traditional hardcopy printed versions, for reading viaan assortment of various e-Reader devices. Various types of e-Readerdevices are available, including portable standalone e-Reader devices ore-Reader software applications that may be loaded on consumer portableelectronic devices or desktop computers.

To enhance the reading experience, various systems have been proposed inwhich audio soundtracks may be played while the user reads an e-book.Some of the systems propose soundtracks that are customised according tothe content of the e-book. Some of the systems synchronise the playbackof the soundtrack with the user's reading speed by calculating thereading speed at each page turn and then reproducing the remainingsoundtrack for playback at each page turn with a modified duration thatis based on the calculated reading speed. Other systems provide asoundtrack for each page of the e-book and then modify the duration ofthe soundtracks based on the user's calculated reading speed.

The above such systems suffer various drawbacks, including deficienciesin the accuracy of the synchronization, playback processinginefficiencies, and lack of flexibility in adapting to various userinteractions with the e-book, such as the altering of font size displaysettings.

In this specification where reference has been made to patentspecifications, other external documents, or other sources ofinformation, this is generally for the purpose of providing a contextfor discussing the features of the invention. Unless specifically statedotherwise, reference to such external documents is not to be construedas an admission that such documents, or such sources of information, inany jurisdiction, are prior art, or form part of the common generalknowledge in the art.

It is an object of the present invention to provide an improved methodand system for making and playing soundtracks that are configured forsynchronized playback with the reading of a corresponding electronictext, or to at least provide the public with a useful choice.

SUMMARY OF THE INVENTION

In a first aspect, the invention broadly consists in a method forcontrolling the commencement of playback of a soundtrack for anelectronic text, the soundtrack comprising multiple audio regionsconfigured for playback during corresponding text regions of theelectronic text, each audio region being defined by audio datacomprising: an audio track for playback in the audio region; a starttext position in the electronic text corresponding to where the playbackof the audio track is to begin; and a stop text position in theelectronic text corresponding to where playback of the audio track is tocease, comprising: receiving an arbitrary commencement text positionwithin the electronic text at which the soundtrack is to commenceplayback; processing the audio data of the soundtrack to identify anyone or more active audio regions within the soundtrack at thatcommencement text position; calculating respective offset cue times forthe audio tracks of any active audio regions based on the respectiveoffset positions of the commencement text position relative to the starttext positions of the audio regions and a user reading speed variableindicative of the user's reading speed; and initiating playback of theaudio tracks of any active audio regions at their respective calculatedoffset cue times to commence soundtrack playback.

In one form, the electronic text may have a designated text index basedon the number of words or characters in the electronic text and thestart text position, stop text position, and commencement text positionare represented by a start index value, a stop index value, and acommencement index value, respectively.

In one form, initiating playback may comprise initiating playback of all(if there are more than one) active audio regions at the commencementtext position simultaneously at their respective offset cue times.

In one form, receiving a commencement text position comprises sensing auser manual selection input of the commencement text position within theelectronic text displayed on a display screen. By way of example,receiving a commencement text position may comprise displaying theelectronic text on a touch screen display and detecting a user manualselection input in the form of a touch input on the touch screen displayin the vicinity of the word or character corresponding to thecommencement text position.

In one form, processing the audio data may comprise searching the audiodata for audio regions in which the commencement text position fallsbetween the start text position and stop text position of the audioregions.

In a second aspect, the invention broadly consists in a system fordisplaying electronic text and controlling playback of a soundtracksynchronized to the reading speed of a user comprising: a displayconfigured to display the electronic text; a user interface configuredto receive user input to control the system; an audio output system thatis operable to generate audio output for the user to hear; an electronictext display module configured to process electronic text datarepresenting the electronic text and control display of the electronictext on the display in response to user input from the user interface; areading position module configured to generate a reading positioncounter corresponding to the estimated current text position in theelectronic text that the user is currently reading, the reading positioncounter incrementing at a rate based at least partly on a configurableuser reading speed variable representing the user's reading speed; and asoundtrack playback module configured to process soundtrack datarepresenting the soundtrack to control playback of the soundtrack overthe audio output system, the soundtrack data defining multiple audioregions for the soundtrack, each audio region being defined by audiodata comprising: an audio track for playback in the audio region; astart text position in the electronic text corresponding to where theplayback of the audio region is to begin; and a stop text position inthe electronic text corresponding to where the playback of the audioregion is to cease; and wherein the soundtrack playback module isconfigured to coordinate playback of the audio regions based on thereading position counter to synchronize playback of the audio regions ofthe soundtrack with the user's reading of the electronic text, andwherein the soundtrack playback module is further operable to commenceplayback of the soundtrack at any arbitrarily user-selected commencementtext position within the electronic text by triggering each audio trackof any one or more active audio regions at the selected commencementtext position to playback at a respective offset cue time from the startof the audio track that is calculated based on the respective offsetposition of the commencement text position relative to the start textposition of the audio region and the user reading speed variable.

In a third aspect, the invention may broadly consist in a method fordisplaying electronic text and synchronizing the playback of asoundtrack for the electronic text, the soundtrack comprising multipleaudio regions configured for playback during corresponding text regionsof the electronic text, each audio region being defined by audio datacomprising: an audio track for playback in the audio region, a starttext position in the electronic text corresponding to where the playbackof the audio track is to begin; and a stop text position in theelectronic text corresponding to where playback of the audio track is tocease, and at least some audio regions further each being defined byplayback effect data that defines one or more controllable audioplayback effect settings that are to be varied across the audio regionor a portion of the audio region based on the user's real-time textposition, comprising: displaying at least a portion of the electronictext on a display; maintaining a reading position counter correspondingto the estimated text position in the electronic text that the user iscurrently reading, the counter incrementing at a rate based on aconfigurable user reading speed variable representing the user's readingspeed; controlling playback of the audio regions of the soundtrack overan audio output system based on the audio data and reading positioncounter to synchronise playback of the audio regions with the user'sestimated reading position; and controlling audio playback effectsettings of the audio output system in real-time based on the readingposition counter for any audio regions having pre-defined playbackeffect data such that the audio playback effect settings are varied inreal-time based on the user's real-time text position.

In one form, the audio output system may comprise a plurality ofchannels such that any overlapping audio regions can playbackconcurrently on separate channels, and wherein the method comprisescontrolling independently the playback effect settings of each channelbased on any pre-defined audio playback effect data associated with theoverlapping audio regions.

In one form, the pre-configured playback effect data may be defined by aprogrammable function the output of which represents a playback effectsetting that is dependent on a variable corresponding to the readingposition counter or a counter that is a product or function of thereading position counter.

In another form, the pre-configured playback effect data associated withan audio region may be in the form of a look-up table which defines adiscrete playback effect setting for a series of discrete text positionsextending across at least a portion of the audio region.

In one form, the pre-configured playback effect data associated with anaudio region may be represented in the form of one or more envelopesthat each define a playback effect setting that is variable across atleast a portion of the audio region, and where the shape of eachenvelope across the audio region or portion of the audio region defineshow its associated playback effect setting is varied across the audioregion or portion of the audio region. In one example, thepre-configured playback effect data may comprise a volume envelope thatdefines a variable volume setting to be applied during playback of theaudio region, and controlling audio playback effect settings of theaudio output system comprises updating a volume setting of the audiooutput system in real-time to match the volume envelope based on theuser's real-time text position within the audio region. Additionally oralternatively, the pre-configured playback effect data may comprise apanning envelope that defines a variable balance setting to be appliedduring playback of the audio region, and controlling the audio playbackeffect settings of the audio output system comprises updating a balancesetting of the audio output system in real-time to match the panningenvelope based on the user's real-time text position within the audioregion.

By way of example only, the pre-configured playback effect data maydefine any one or more of the following controllable audio playbackeffect settings: volume, balance, equalization, reverberation,distortion, and phase-shifting.

In a fourth aspect, the invention broadly consists in a system fordisplaying electronic text and controlling playback of a soundtracksynchronized to the reading speed of a user comprising: a displayconfigured to display the electronic text; a user interface configuredto receive user input to control the system; an audio output system thatis operable to generate audio output for the user to hear and which hascontrollable audio playback effect settings for applying audio playbackeffects to the audio output; an electronic text display moduleconfigured to process electronic text data representing the electronictext and control display of the electronic text on the display inresponse to user input from the user interface; a reading positionmodule configured to generate a reading position counter representingthe estimated current text position in the electronic text that the useris currently reading, the reading position counter incrementing at arate based at least partly on a configurable user reading speed variablerepresenting the user's reading speed; and a soundtrack playback moduleconfigured to process soundtrack data representing the soundtrack tocontrol playback of the soundtrack over the audio output system, thesoundtrack data defining multiple audio regions for the soundtrack, eachaudio region being defined by audio data comprising: an audio track forplayback in the audio region; a start text position in the electronictext corresponding to where the playback of the audio region is tobegin; and a stop text position in the electronic text corresponding towhere the playback of the audio region is to cease; and at least some ofthe audio regions further each being defined by playback effect datathat defines one or more controllable audio playback effect settingsthat are to be varied across an audio region or a portion of the audioregion based on the user's real-time text position, and wherein thesoundtrack playback module is configured to coordinate playback of theaudio regions based on the reading position counter to synchronizeplayback of the audio regions of the soundtrack with the user's readingof the electronic text, and the soundtrack playback module being furtherconfigured to control the audio playback effect settings of the audiooutput system in real-time based on the reading position counter for anyaudio regions having associated playback effect data such that the audioplayback effect settings for each such audio region are varied inreal-time based on the user's real-time text position.

In a fifth aspect, the invention may broadly consist in a method ofupdating a reading speed variable representing the reading speed of auser reading electronic text, the reading speed variable being used forsynchronizing the playback of a soundtrack for the electronic text,comprising: displaying at least a portion of the electronic text on anelectronic display, the display having an associated user input toenable user interaction with the display; sensing and tracking movementof the user input relative to the displayed electronic text, themovement representing the user's reading position on the display as theytrace their reading of the electronic text via the user input;generating data indicative of the quantity of text read and time takenfor the movement; calculating a measurement of reading speed based onthe quantity of text read during the movement and the time taken for themovement; and updating the reading speed variable based on thecalculated measurement of reading speed.

In one form, sensing and tracking movement of a user input may comprisesensing a start text position in the electronic text at which the userinitiates the movement and an end text position in the electronic textat which the user ceases the movement, and generating data indicative ofthe quantity of the text traversed during the movement comprisesdetermining the number of words or characters between the detected startand end text positions associated with the movement.

In one form, generating data indicative of the time taken for themovement may comprises initiating a timer upon initiation of themovement at the start text position and halting the timer at the end ofthe movement at the end text position, the output of the timer beingindicative of the time taken for the movement.

In one form, the electronic display may be a touch screen display havinga user input in the form of touch input upon the surface of the displaywhich generates representative touch signals, and wherein the start textposition in the electronic text is detected at initiation of themovement upon sensing user contact with the surface of the display andthe end text position is sensed at the position in the electronic textwhere the user disengages their contact from the surface of the display.

In one form, sensing and tracking movement of a user input may comprisetracking the user input as it moves substantially horizontally relativeto the displayed electronic text word-by-word. Additionally oralternatively, sensing and tracking movement of the user input maycomprise tracking the user input as it moves substantially verticallyrelative to the displayed electronic text line-by-line.

In one form, calculating a measurement of reading speed may comprisegenerating a measurement of reading speed at the end of the movementbased on the total quantity of text read and total time taken. Inanother form, calculating a measurement of reading speed may comprisecontinuously or periodically generating a measurement of reading speedin real-time based on quantity of text read since the initiation of themovement and the time elapsed since the initiation of the movement; andcontinuously or periodically updating the reading speed variable.

In a sixth aspect, the invention broadly consists in a system fordisplaying electronic text and controlling playback of a soundtracksynchronized to the reading speed of a user comprising: a displayconfigured to display the electronic text; a user interface configuredto receive user input to control the system; an audio output system thatis operable to generate audio output for the user to hear; an electronictext display module configured to process electronic text datarepresenting the electronic text and control display of the electronictext on the display in response to user input from the user interface; areading position module configured to generate a reading positioncounter corresponding to the estimated current text position in theelectronic text that the user is currently reading, the reading positioncounter incrementing at a rate based at least partly on a configurableuser reading speed variable representing the user's reading speed; asoundtrack playback module configured to process soundtrack datarepresenting the soundtrack to control playback of the soundtrack overthe audio output system, the soundtrack data defining multiple audioregions for the soundtrack, each audio region being defined by audiodata comprising: an audio track for playback in the audio region; astart text position in the electronic text corresponding to where theplayback of the audio region is to begin; and a stop text position inthe electronic text corresponding to where the playback of the audioregion is to cease; and wherein the soundtrack playback module isconfigured to coordinate playback of the audio regions based on thereading position counter to synchronize playback of the audio regions ofthe soundtrack with the user's reading of the electronic text; andwherein the system further comprises a reading speed adjustment modulethat is configured to sense and track movement of a user input relativeto the displayed electronic text for a tracking period initiated by theuser and which calculates and updates the user reading speed variablebased on the quantity of text read during the movement and the timetaken for the movement, the movement representing the user's readingposition on the display as they trace their reading of the electronictext via the user input.

In a seventh aspect, the invention broadly consists in a method ofupdating a reading speed variable representing the reading speed of auser reading electronic text, the reading speed variable being used forsynchronizing the playback of a soundtrack for the electronic text,comprising: displaying at least a portion of the electronic text on adisplay; detecting a first user input indicative of a first reading textposition corresponding to the user's arbitrary reading text position inthe electronic text at the time of the first input, and generating dataindicative of the first reading text position; detecting a secondsubsequent user input indicative of a second reading text positioncorresponding to the user's arbitrary reading position in the electronictext at the time of the second input, and generating data indicative ofthe second reading text position; calculating the quantity of text readbetween the first and second reading text positions based on thegenerated data; calculating an estimate of the user's reading speedbased on the calculated quantity of text read and the time periodbetween the first and second user inputs; and updating the reading speedvariable based on the calculated estimate of the user's reading speed.

The term ‘arbitrary’ in this context is intended to mean that the usermay indicate their current reading position at any desired text positionin the electronic text, and at any time during reading of the electronictext.

In one form, the electronic text may be indexed based on the number ofwords or characters in the electronic text, and the data indicative ofthe first reading text position comprises a first index valuecorresponding to the index value at the first reading text position, andthe data indicative of the second reading text position comprises asecond index value corresponding to the index value at the secondreading text position, and wherein calculating the quantity of text readcomprises calculating the absolute difference between the first andsecond index values.

In one form, the method may further comprise generating a first timestamp and a second time stamp corresponding to the respective times ofthe first user input and second user input, and calculating the timeperiod between the first and second user inputs based on absolutedifference between the first and second time stamps. In another form,the method may further comprise starting a timer on detection of thefirst user input and stopping the timer on detection of the second userinput, and the output of the timer representing the time period betweenthe first and second user inputs.

In one form, the display may be a touch screen display, and detectingthe first and second user inputs comprises sensing user selections ofthe arbitrary text positions in the electronic text by sensing a usertouch input on the surface of the touch screen display at locationscorresponding to the arbitrary text positions. In another form,detecting the first and second user inputs may comprise detecting userselection of the arbitrary text positions in the electronic text bysensing user activation via a user input device of a pointer on thedisplay situated at the location corresponding to the arbitrary textpositions.

In one form, the method may further comprise calculating a new estimateof the user's reading speed for each new subsequent user inputindicative of the user's arbitrary reading text position in theelectronic text, each new estimate of the user's reading speed beingcalculated based on the quantity of text and time period between theuser input indicative of the newest arbitrary reading text position andthe previous indicated arbitrary reading text position.

In one form, the method may comprise updating the user reading speedvariable by applying an update algorithm for each new calculatedestimate of the user's reading speed, the update algorithm comprising:calculating a new moving average of the reading speed variable based ona predetermined number of previous reading speed variables; calculatinga difference value indicative of the percentage change between the newcalculated estimate of the user's reading speed and the moving average;determining a probability weighting for the new calculated estimate ofthe user's reading speed based on the difference value and a probabilitydensity function; and calculating a new reading speed variable based onthe new calculated estimate of the user's reading speed, the movingaverage, and the probability weighting.

In an eighth aspect, the invention broadly consists in a system fordisplaying electronic text and controlling playback of a soundtracksynchronized to the reading speed of a user comprising: a displayconfigured to display the electronic text; a user interface configuredto receive user input to control the system; an audio output system thatis operable to generate audio output for the user to hear; an electronictext display module configured to process electronic text datarepresenting the electronic text and control display of the electronictext on the display in response to user input from the user interface; areading position module configured to generate a reading positioncounter corresponding to the estimated current text position in theelectronic text that the user is currently reading, the reading positioncounter incrementing at a rate based at least partly on a configurableuser reading speed variable representing the user's reading speed; asoundtrack playback module configured to process soundtrack datarepresenting the soundtrack to control playback of the soundtrack overthe audio output system, the soundtrack data defining multiple audioregions for the soundtrack, each audio region being defined by audiodata comprising: an audio track for playback in the audio region; astart text position in the electronic text corresponding to where theplayback of the audio region is to begin; and a stop text position inthe electronic text corresponding to where the playback of the audioregion is to cease; and wherein the soundtrack playback module isconfigured to coordinate playback of the audio regions based on thereading position counter to synchronize playback of the audio regions ofthe soundtrack with the user's reading of the electronic text; and thesystem further comprising a reading speed adjustment module that isconfigured to: detect a first user input indicative of a first readingtext position corresponding to the user's arbitrary reading textposition in the electronic text at the time of the first input, andgenerate data indicative of the first reading text position; detect asecond subsequent user input indicative of a second reading textposition corresponding to the user's arbitrary reading position in theelectronic text at the time of the second input, and generate dataindicative of a second reading text position; calculate the quantity oftext read between the first and second reading text positions based onthe generated data; calculate an estimate of the user's reading speedbased on the calculated quantity of text read and time period betweenthe first and second user inputs; and update the configurable userreading speed variable based on the calculated estimate of the user'sreading speed.

In a ninth aspect, the invention broadly consists in a method ofupdating a reading speed variable representing the reading speed of auser reading electronic text, the reading speed variable being used forsynchronizing the playback of a soundtrack for the electronic text,comprising: displaying at least a portion of the electronic text on adisplay; displaying a moving reading position indicator on the displaywhich identifies the current text position that is estimated as beingread based on an initial reading speed variable representing the user'sestimated reading speed, the reading position indicator moving relativeto the text at a rate based on the reading speed variable; providing auser adjustment interface that is operable to increase or decrease thereading speed variable; and updating the reading speed variableaccording to any user adjustment to thereby modify the rate of movementof the reading position indicator to match the updated reading speedvariable, and wherein the user adjustment interface is operablesimultaneously as the electronic text and moving reading positionindicator are displayed such that the user can view in real-time theeffect of any adjustment of the reading speed variable.

In one form, the user adjustment interface may be in the form ofactivatable increase and decrease inputs that are configured torespectively increase or decrease the reading speed variable by apredetermined value upon activation by a user. In another form, the useradjustment interface may be in the form of an operable dial or sliderthat is moveable to increase or decrease the reading speed variable to auser selected value within a scale of reading speed values definedbetween upper and lower limits.

In one form, the reading position indicator may be an inline indicatorthat is configured to traverse the lines of text word-by-word orcharacter-by-character at a rate that is dependent on the reading speedvariable. Additionally or alternatively, the reading position indicatormay be a margin indicator that is configured to move down the margin ofthe display text line-by-line at a rate that is dependent on the readingspeed variable.

In a tenth aspect, the invention broadly consists in a system fordisplaying electronic text and controlling playback of a soundtracksynchronized to the reading speed of a user comprising: a displayconfigured to display the electronic text; a user interface configuredto receive user input to control the system; an audio output system thatis operable to generate audio output for the user to hear; an electronictext display module configured to process electronic text datarepresenting the electronic text and control display of the electronictext on the display in response to user input from the user interface; areading position module configured to generate a reading positioncounter corresponding to the estimated current text position in theelectronic text that the user is currently reading, the reading positioncounter incrementing at a rate based at least partly on a configurableuser reading speed variable representing the user's reading speed; asoundtrack playback module configured to process soundtrack datarepresenting the soundtrack to control playback of the soundtrack overthe audio output system, the soundtrack data defining multiple audioregions for the soundtrack, each audio region being defined by audiodata comprising: an audio track for playback in the audio region; astart text position in the electronic text corresponding to where theplayback of the audio region is to begin; and a stop text position inthe electronic text corresponding to where the playback of the audioregion is to cease; and wherein the soundtrack playback module isconfigured to coordinate playback of the audio regions based on thereading position counter to synchronize playback of the audio regions ofthe soundtrack with the user's reading of the electronic text; and thesystem further comprising a reading speed adjustment module that isconfigured to: display a moving reading position indicator on thedisplay which identifies the estimated current text position that isestimated as being read based on the reading position counter; provide auser adjustment interface that is operable to increase or decrease theconfigurable user reading speed variable; and update the user readingspeed variable according to any user adjustment to thereby modify therate of movement of the reading position indicator, and wherein the useradjustment interface is operable simultaneously as the electronic textand moving reading position indicator are displayed such that the usercan view in real-time the effect of any adjustment of the user readingspeed variable.

In another aspect, the invention may broadly consist in a method ofmaking a soundtrack for synchronised playback with the reading of anelectronic text comprising the steps of: designating a text index to theelectronic text based on the number of words or characters in theelectronic text; defining multiple audio regions for the soundtrack,each audio region being defined by configurable audio data and the audiodata being configured by: selecting an audio track from a database forplayback in the audio region; and designating a start index valuerepresenting the start text position in the electronic textcorresponding to where the playback of the audio region is to begin anda stop index value representing the stop text position in the electronictext corresponding to where the playback of the audio region is tocease; and compiling the audio data associated with the audio regionsinto an output data file representing the soundtrack.

In some embodiments, the text index is based on the number of words inthe electronic text. Each word in the electronic text may be designateda unique index value that represents its position in the text relativeto the other words. By way of example, each word may have an index valuethat corresponds to its word number or word count in the electronictext. In these embodiments, the start and stop text positions representstart and stop words in the electronic text where the audio region is tobegin and cease, respectively. In other embodiments, the text index maybe based on the number of characters, and the index values maycorrespond to a character number or character count in the electronictext.

In some embodiments, the step of configuring the audio data for an audioregion further comprises defining an audio type for the audio regionfrom a list of audio types. By way of example, the types may comprisebackground music, ambient sounds, and sound effects.

In some embodiments, the step of configuring the audio data for an audioregion further comprises defining a channel from a selection of channelsover which the audio track of the audio region will playback.Preferably, the soundtrack comprises multiple channels such that two ormore overlapping audio regions may playback concurrently.

In some embodiments, the step of compiling audio data into an outputdata file representing the soundtrack further comprises adding headerdata to the output data file comprising information indicative ofelectronic text to which the soundtrack corresponds and/or generalsoundtrack identification information.

In some embodiments, the step of compiling the audio data into theoutput data file representing the soundtrack further comprises addingelectronic text data representing the electronic text into the outputdata file such that the output data file is a multimedia data filecomprising both data representing the soundtrack and the correspondingelectronic text.

In some embodiments, the audio data for each audio region comprises theaudio track in the form of an actual audio track sound file forplayback, i.e. in such embodiments the output data file of thesoundtrack comprises all data required for soundtrack playback,including the audio files of the audio tracks. In other embodiments, theaudio data for each audio region comprises a link (e.g. filepath or fileaddress or location) to the actual audio track sound file for playback.

In another aspect, the invention may broadly consist in a system fordisplaying electronic text having a designated text index based on thenumber of words or characters in the electronic text and controllingplayback of a soundtrack synchronized to the reading speed of a usercomprising: a display configured to display the electronic text; a userinterface configured to receive user input to control the system; anaudio output system that is operable to generate audio output for theuser to hear; an electronic text display module configured to processelectronic text data representing the electronic text and controldisplay of the electronic text on the display in response to user inputfrom the user interface; a reading position module configured togenerate a reading position counter representing an index valuecorresponding to the estimated current text position in the electronictext that the user is currently reading, the reading position counterincrementing at a rate based at least partly on a configurable userreading speed variable representing the user's reading speed; and asoundtrack playback module configured to process soundtrack datarepresenting the soundtrack to control playback of the soundtrack overthe audio output system, the soundtrack data defining multiple audioregions for the soundtrack, each audio region being defined by audiodata comprising: an audio track for playback in the audio region; astart index value representing the start text position in the electronictext corresponding to where the playback of the audio region is tobegin; and a stop index value representing the stop text position in theelectronic text corresponding to where the playback of the audio regionis to cease; and wherein the soundtrack playback module is configured tocoordinate playback of the audio regions based on the reading positioncounter to synchronize playback of the audio regions of the soundtrackwith the user's reading of the electronic text.

In some embodiments, the electronic text is indexed based on the numberof words in the electronic text. For example, each word is designated anindex value corresponding to its word number or word count in theelectronic text. In such embodiments, the reading position counterrepresents the index value of the estimated current word being read. Inother embodiments, the text indexing may be based on the number ofcharacters, and the reading position counter represents the index valueof the estimated current character being read.

In some embodiments, the user interface comprises a re-synchronizationinterface that is configured to receive user input representing are-sync text position corresponding to the desired or actual textposition in the electronic text to which re-synchronisation of thesoundtrack playback should occur. By way of example, the user input maybe in the form of a user identifying an actual word or character in theelectronic text that is to be the re-sync text position. In oneembodiment, the display comprises a touch-screen interface and there-synchronization interface is configured to monitor touch-screeninteraction for a user indicator input representing the actual wordbeing read. In one example, the input indicator may be in the form of adouble-tap or single-tap or pressing on the actual word or charactercorresponding to the desired re-sync text position. In another example,the input indicator may be in the form of the user dragging a readingposition indicator (′inline′ or ‘margin’) to the actual word orcharacter corresponding to the desired re-sync text position. There-synchronisation interface is configured to update or modify thereading position counter to match the index value corresponding to theword or character at the re-sync text position such that the soundtrackplayback module halts and re-initialises playback of the soundtrack andany active audio regions at the re-sync text position.

The soundtrack playback module may also be configured to controlconcurrent playback of any two or more overlapping audio regions in thesoundtrack over their respective channels.

In another aspect, the invention may broadly consist in a method fordisplaying electronic text having a designated text index based on thenumber of words or characters in the electronic text and synchronizingthe playback of a soundtrack for the electronic text according to auser's reading speed comprising: displaying at least a portion of theelectronic text on the display; maintaining a reading position counterrepresenting an index value corresponding to the estimated current textposition (word or character) in the electronic text that the user iscurrently reading; incrementing the reading position counter at a rateat least partly based on a configurable user reading speed variablecorresponding to the user's estimated reading speed; and controllingplayback of the soundtrack based on the reading position counter, thesoundtrack being defined by multiple audio regions for the soundtrack,each audio region being defined by audio data comprising: an audio trackfor playback in the audio region; a start index value representing thestart text position in the electronic text corresponding to where theplayback of the audio track is to begin; and a stop index valuerepresenting the stop text position in the electronic text correspondingto where playback of the audio track is to cease.

In one form, controlling the playback of the soundtrack comprisesinitiating playback of an audio region when the reading position countermatches the start index value of that audio region, and ceasing playbackof the audio region when the reading position counter matches the stopindex value of that audio region. More preferably, the method comprisescontrolling playback of any two or more overlapping audio regions forconcurrent playback.

In another aspect, the invention may broadly consist in a method ofupdating a reading speed variable representing an estimate of thereading speed of a user reading electronic text, the electronic textbeing indexed based on the number of words or characters in theelectronic text, comprising: displaying at least a portion of theelectronic text on a display; detecting each new user input indicativeof a new reading text position corresponding to the user's arbitraryreading text position in the electronic text at the time of the new userinput; calculating a new estimate of the user's reading speed for eachnew user input based on the length of text and time period between thenewest reading text position and the previous indicated reading textposition; and updating the reading speed variable by applying an updatealgorithm to each new calculated estimate of the user's reading speed.

Each of the aspects of the invention above may have any one or more ofthe features mentioned in the respect of the other aspects of theinvention. In particular, it will be appreciated that the featuresmentioned above may be interchanged and combined in various forms, aswill further be evident from the following description of exampleembodiments.

The phrase “text position” as used in this specification and claims,unless the context suggests otherwise, is intended to refer to aposition or location within the electronic text such as typically a lineor word or character of the electronic text or any other arbitraryselected position with the electronic text, and the position may bereferenced or identified in any suitable manner appropriate to theelectronic text, whether indirectly or directly, including but notlimited to: an index value, numeric or otherwise, corresponding to theindex position of a line, word or character in the electronic text foran electronic text that has been indexed, including a line number, wordnumber of character number for example; or any other positionalreference scheme for identifying locations within electronic text.

The term “comprising” as used in this specification and claims means“consisting at least in part of”. When interpreting each statement inthis specification and claims that includes the term “comprising”,features other than that or those prefaced by the term may also bepresent. Related terms such as “comprise” and “comprises” are to beinterpreted in the same manner.

As used herein the term “and/or” means “and” or “or”, or both.

As used herein “(s)” following a noun means the plural and/or singularforms of the noun.

The invention consists in the foregoing and also envisages constructionsof which the following gives examples only.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the invention will be described by way ofexample only and with reference to the drawings, in which:

FIG. 1 shows a schematic block diagram of a soundtrack creation systemin accordance with an embodiment of the invention;

FIG. 2 shows a flow diagram of a soundtrack production method inaccordance with an embodiment of the invention;

FIG. 3 a shows a screenshot of a graphical user interface (GUI) of asoundtrack creation system in accordance with an embodiment of theinvention;

FIG. 3 b shows a close up of the main toolbar indicated at A in FIG. 3a;

FIG. 4 shows a screenshot of the GUI of the soundtrack creation systemdisplaying an imported electronic text file;

FIG. 5 shows a flow diagram for the process of configuring the audiodata for an audio region of the soundtrack in accordance with anembodiment of the invention;

FIG. 6 shows a screenshot of the GUI of the soundtrack creation systemand the selection of a text region corresponding to a new audio region;

FIG. 7 shows a screenshot of a GUI of the soundtrack creation system forconfiguring the audio data of a new audio region of the soundtrack;

FIG. 8 shows a screenshot of a GUI for designating the audio type of anew audio region of the soundtrack;

FIG. 9 shows a screenshot of a GUI for configuring a volume envelope tobe applied to an audio region of the soundtrack;

FIG. 10 shows a screenshot of a GUI for configuring a panning envelopeto be applied to an audio region of the soundtrack;

FIG. 11 shows a screenshot of a GUI for adjusting the global playbackvolume of audio regions in the soundtrack having a particular audiotype;

FIG. 12 shows a screenshot of a GUI for creating a new audio type foraudio regions of the soundtrack;

FIG. 13 shows a screenshot of the GUI of FIG. 4 in which multiple audioregions have been applied to an imported electronic text file in apartially created soundtrack;

FIG. 14 shows a screenshot of an audio data table containing the audiodata for multiple audio regions of a partially created soundtrack;

FIG. 15 shows a schematic block diagram of a user playback system fordisplaying electronic text and for controlling playback of a soundtrackin accordance with an embodiment of the invention;

FIG. 16 shows a screenshot of a GUI of the user playback systemdisplaying electronic text with a first type of in-line reading positionindicator;

FIG. 17 shows the GUI of FIG. 16 but with a second type of in-linereading position indicator;

FIG. 18 shows the GUI of FIG. 16 but with a margin reading positionindicator;

FIG. 19 shows a screenshot of the GUI of FIG. 16 with a tools menuinterface activated;

FIG. 20 shows the GUI of FIG. 16 with an audio settings interfaceactivated;

FIG. 21 shows the GUI of FIG. 16 with a font settings interfaceactivated;

FIG. 22 shows a flow diagram of an automatic adjustment method forupdating a user reading speed variable based on arbitrary readingposition inputs from the user in accordance with an embodiment of theinvention;

FIG. 23 shows a flow diagram of a method for calculating a new estimateof the user's reading speed based on successive arbitrary readingposition inputs from the user within the automatic adjustment method ofFIG. 22 in accordance with an embodiment of the invention; and

FIG. 24 shows a flow diagram of a method of updating a user readingspeed variable based on a new estimate of the user's reading speedwithin the automatic adjustment method of FIG. 22 in accordance with anembodiment of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 1. Overview

The invention relates to a method and system for making and playingsoundtracks for electronic text, such as electronic books (the e-books).The invention may be applied to electronic text in any format orfile-type, including but not limited to those associated with thefollowing file extensions: aeh, djvu, epub, fb2, html, xhtml, azw, lit,prc, mobi, exe, pdb, txt, pdb, pdf, ps, rgo, pdb, tr2, tr3, wol, rtf,doc, docx, asc, lit, wri, odt, text, pwi, rtx, sdw, strings, utf8,utf16, wp*, wpw, wri, wtx, asc, csv, lwp, dtp, indd, pmd, ppp, pub, fin,and any type of mark up language types, including but not limited toSGML, HTML, XHTML, HTML5, XML, and LaTex. The electronic text maycomprise text alone, or may be provided in combination with imagery,whether static or moving (e.g. video or other animations).

The soundtrack is customised to the electronic text and in particularcustomized to the content of the electronic text to enhance the readingexperience for a reader of the electronic text. The soundtrack maycomprise audio tracks of different types and may be configured to playconcurrently, sequentially, or both while the user is reading theelectronic text. Playback of the soundtrack is synchronized with thereading speed of the user. By way of example, the electronic text may bea fiction or non-fiction book, and the soundtrack may comprise acombination of different audio types, including ambient noise, soundeffects, and background music that may be configured to play back at aspecific point or over specific portions of the electronic text in asynchronized manner as the user is reading those portions of theelectronic text.

The soundtrack comprises multiple audio regions that are designated toplayback during designated text portions of the electronic text. Eachaudio region comprises audio data defining the audio track to playbackand audio properties defining the desired nature of the playback. Theaudio data specifies a start text position in the electronic text atwhich the audio region is to commence playback and a stop text positionat which the audio region is to cease playback. The electronic text isdesignated a numerical text index based on the number or words orcharacters in the electronic text, and the start and stop text positionsare defined by corresponding start and stop index values respectively.In one embodiment, the indexing of the text is based on the number ofwords in the electronic text such that each word has a unique indexvalue representing its position in the electronic text. The index valuesfor each word may correspond to their word number or word count in theelectronic text. In such embodiments, the audio regions playback betweenstart and stop index values representing the selected start and stopwords in the electronic text for that audio region.

When the user is reading the electronic text, a reading position counteror identifier is maintained which corresponds to text position in theelectronic text at which the user is estimated as currently reading. Thereading position counter increments or changes at a rate at leastpartially based on the user's reading speed. This reading positioncounter allows the playback of the audio regions of the soundtrack to besynchronized with the user's reading speed such that playback of anaudio region is triggered when the reading position counter matches thestart index value of the start word for the audio region, and likewiseceases when the reading position counter matches the stop index value ofthe stop word of the audio region.

The sound track comprises multiple audio regions and playback of two ormore regions may partially or entirely overlap such that two or moreaudio regions can playback concurrently if desired. The soundtrack isprovided with multiple channels for playback and overlapping audioregions are designated different channels in the set of availablechannels to allow for concurrent playback. This enables different audiotracks, such as sound effects, ambient noise, or background music to beconcurrently played and mixed together to provide an overall soundtrackthat enhances the user's reading experience for that particularelectronic text.

A more detailed explanation of an embodiment of a system (soundtrackcreation system) and method for producing a soundtrack for an electronictext, and a system (user playback system) and method for displaying theelectronic text and controlling playback of the soundtrack will now beprovided by way of example only.

2. A System and Method for Producing a Soundtrack Soundtrack CreationSystem

In this embodiment, the soundtrack creation system is provided as anapplication program or software running on an electronic hardware systemsuch as a personal computer although it will be appreciated that thecreation system may be provided or implemented in any suitable form,whether in software, hardware, or a combination of these. Thisembodiment of the soundtrack creation system and method will bedescribed with reference to FIGS. 1-14.

Referring to FIG. 1, an example of the hardware system 100 upon whichthe soundtrack creation system software may run as shown. The hardwaresystem 100 includes a processor 102 for executing software orcomputer-readable instructions, memory 104 for data processing andexecution of instructions, data storage 106 such as a hard disk drivefor data storage, a user interface 108 to enable the user to interactwith the hardware system and software, a display 110 for displayinggraphical user interfaces (GUIs) of the soundtrack creation software,and an audio output system 112 for producing audio output over speakersor headphones. The hardware system 100 may also comprise external deviceinterfaces and/or communication modules for establishing communicationlinks 114 to external networks, such as the internet or an extranet orintranet 116 or communication links 118 to external databases 120 suchmusic libraries. It will be appreciated this hardware system may beprovided in the form of a personal computer, whether desktop, laptop,notebook or any other electronic hardware platform or system.

Soundtrack Production Process

Referring to FIG. 2, a typical soundtrack production process will now bedescribed by way of example only. The soundtrack production process 200for generating a soundtrack for electronic text commences at 202 byloading or importing an electronic text file containing electronic textdata representing the electronic text into the soundtrack creationsystem. By way of example, the soundtrack creation system may providethe user with a graphical user interface (GUI) 300 as shown in FIG. 3 a.FIG. 3 b shows a close up of a main toolbar 302 of the GUI 300. The maintoolbar 302 comprises various menus, interfaces and quick-launch buttonsfor the user to interact with when creating the soundtrack. Inparticular, buttons 304 is provided for opening and saving soundtrackdata files. Buttons 306 is provided for playing, pausing and stoppingpreview playback of a soundtrack. Button 308 is provided to access aninterface for editing the playback properties of audio region types.Button 310 is provided for providing an interface for configuring audiovolumes for audio region types. Button 312 is provided for assessing aninterface for muting or disabling selected audio region types. Interface314 is provided as a text searching tool for words or phrases within theelectronic text. Interface 316 is provided for setting a reading speedfor the preview of the soundtrack playback, which may be a reading speedvariable value defined in words per minute (wpm).

Reverting to FIG. 2, once the soundtrack creation system is initiated, ahome screen is displayed as shown in FIG. 3 a which includes the toolbar302 and a text display area or region 318 to create a new soundtrack foran electronic text. The electronic text file, for example an e-book 204,must be imported or loaded into the system from local or external datastorage to the system. As previously described, the electronic text filemay be in any available format for electronic text, including by way ofexample only epub, pdf, rtf, and txt, or any of the other formatsmentioned above. Once imported, the e-book file comprising theelectronic text data may be subjected to optional format processing 204depending on the nature of the file format imported. Once formatted, theelectronic text is displayed in the display area 318 of the GUI 300 asshown in FIG. 4.

The system is configured to apply or designate a text index 206 to theelectronic text displayed based on the number of words or characters inthe electronic text. This embodiment will be described in the context ofa word-based indexing scheme, but it will be appreciated that acharacter-by-character based indexing scheme could alternatively beused. The word indexing applies a unique index value to each word in theelectronic text. The particular word indexing scheme selected may vary,but the index value associated with each word in the electronic textrepresents the position of the word relative to the remaining words inthe electric text. In this embodiment, a word indexing scheme is appliedin which each word has an index value that corresponds to the integerword number or word count associated with that word's position in theelectronic text. For example, the first word has an index value ‘1’, thesecond word has an index value ‘2’ and so on to the last word in theelectronic text. However, it will be appreciated that any other wordnumbering or offset index values relative to a reference point could beapplied in alternative embodiments if desired. After indexing, multipleaudio regions for the soundtrack are defined 208. As previouslydescribed, the soundtrack comprises multiple audio regions, each definedfor playback in a text region designated between a start word and an endword in the electronic text. The playback position and propertiesconfigured for each audio region are defined by audio data or metadataassociated with each audio region.

Referring to FIG. 5, the process of defining the audio regions of thesoundtrack will be described in more detail. It will be appreciated thatthe order of the steps is not necessarily essential, and some steps areoptional. Each audio region can be considered as being configured forplayback over the duration of a corresponding text region in theelectronic text that is defined between a start word and end word forthe audio region. Each new audio region may be created by initiallyselecting a text region 400 in the electronic text to which the audioregion is to apply. By way of example with reference to FIG. 6, in thisembodiment of the soundtrack creation system, the text region may beselected by highlighting the text region 320 in the text display area318. As shown, the text region has a start word indicated at 322 and astop word indicated at 324. Once a text region is selected, thesoundtrack creation system is configured to generate a new audio regionconfiguration interface 340 shown in FIG. 7 by way of example. The audioregion configuration interface 340 is automatically populated with thestart word index value 342 and stop word index value 344. In thisembodiment, the start word index value 342 corresponds to the wordnumber of start word 322 in the electronic text and the stop word indexvalue 344 corresponds to the word number at the stop word 324 in theelectronic text. Additionally or alternatively, it will be appreciatedthat a new audio region configuration interface may be initiated withblank start and stop fields 342,344 which may then be manuallyconfigured. Additionally or alternatively, the user may simply highlighta start word in the text display area 318 to automatically populate thestart word index value 342 and then may manually adjust the stop wordindex value as desired.

The remaining steps of configuring the audio data 402 for an audioregion will now be described in further detail with reference to FIGS. 5and 7. It will be appreciated that not all the various audio propertiesfor the audio region need to be configured depending on the desiredaudio playback properties for the audio region and also the order of theconfiguration steps is not essential and may be modified.

An audio type 404 for the audio region is selected as shown at 346 inFIG. 7. Referring to FIG. 8, the audio type for the audio region may beselected from a list of audio types 350 provided in audio region typeinterface 352. In this embodiment, there are three audio types, namely‘music’, ‘effect’, and ‘ambient’ as will be explained in further detaillater. Each audio region type may have its own global audio playbackproperties and these properties may be edited. Additionally, audio typesmay be deleted or added as required. There may be one or a plurality ofselectable audio types depending on the soundtrack being configured. Theselected audio type is intended to define the nature of the audioregion. For example, if an audio region is for background music, thenthe audio type selected will be ‘music’. If the audio region is a soundeffect, then the ‘effect’ audio type is selected. If the audio region isintended to be an ambient sound or noise, then the ‘ambient’ audio typeis selected.

A channel 406 from a set of available channels is selected over whichthe audio region will playback and as shown at 348 in FIG. 7. In thisembodiment, the soundtrack is provided with at least two channels andpreferably more channels to enable playback of the audio regions tooverlap. Overlapping audio regions are configured to have differentchannels. By way of example, the soundtrack may provide six channelssuch that up to six audio regions may overlap at any one time forplayback in the soundtrack. However, it will be appreciated that inalternative embodiments a single channel may be provided for soundtracksin which there are no overlapping audio regions such as for soundtrackscomprising only sequential or spaced apart audio regions of backgroundmusic or soundtrack comprising non-overlapping sound effects, ambientnoise and background music. The soundtrack creation system may beconfigured to automatically select or designate different channels tonew audio regions that have text regions that overlap with those ofexisting audio regions.

Optionally, the start and stop index values 342, 344 as shown in FIG. 7may be manually adjusted 408 to modify the start and stop words betweenwhich the playback of the audio region begins and ceases respectivelyfor the audio region if desired.

An audio track is selected 410 for playback in the audio region as shownat 354. The audio track may be an audio file that is selected from anaccessible database or music library stored locally or externally to thesystem. The database or music library may comprise a range of music,sound effects, ambient noise audio files, or any other audio track thatmay be selected to enhance the reading experience throughout theduration of the electronic text by providing sound effects that matchevents occurring the electronic text or ambient sounds or music thatreflects the mood or atmosphere during passages or portions of theelectronic text, by way of example only. The audio file may be providedin any sound file format including but not limited to audio files havingthe following file extensions: MP3, WAV, AAC, WMA, MP4, M4A, OGG, CAF,AIFF, RAW, FLAC, TTA, GSM, OTS, SWA, VOX, MOD, PSF, RMJ, SPC, andMusicXML. It will be appreciated that the audio track selection may be afile path or link to the destination of the audio file in the databaseor music library for the audio track selected. It will be appreciatedthat the natural duration of the audio track selected for one type ofaudio region may vary compared to audio tracks selected for another typeof audio region. Typically, specific audio types may have similarduration ranges. For example, ‘background music’ audio region types maytypically have audio tracks having durations of 2-7 minutes, whereas‘sound effects’ may have durations of a few seconds or milliseconds byway of example only.

Selecting an audio track 410 may optionally comprise also selecting apreview audio track as shown in the selection field 356 in FIG. 7. Inone embodiment, the preview audio track may be a lower quality andsmaller file version than the primary audio track 354 that will beincluded in the eventual soundtrack data file. For example, thesoundtrack creation system may be configured to cue the preview audiotrack when the user wishes to test the timing and sequencing of theplayback of specific audio regions during the soundtrack creation,rather than the system having to load and playback the full high qualityaudio track 354. In another embodiment, the preview audio track at 356may correspond to the primary audio track converted into a file formatthat corresponds to the most suitable or desired audio playback filetype of the target hardware device or system upon which the userplayback system will be implemented for playback of the soundtrack. Forexample, the primary audio track may be selected from a database of wayfile types, and the preview audio track may link to the correspondingaudio track in an mp3 format, which may be the preferred file format fora particular target user playback system. It will be appreciated thatthe preview audio track may be automatically converted from the primaryaudio track file type or that the database may comprise each audio trackin multiple formats for selecting and compiling into the soundtrack. Inoperation, the soundtrack creation system may be configured toautomatically compile the soundtrack data file with audio track links oraudio track files in a format corresponding to a desired target audiofile format for the user playback system, by either converting theprimary audio tracks into the target file format, or by selecting thecorresponding audio tracks in the target file format. It will beappreciated that the soundtrack creation system may be operated toproduce or provide the same soundtrack in data files comprisingdifferent desired file formats for the audio tracks to suit differenttarget hardware systems for the user playback system.

An individual playback volume 412 may be configured for each audioregion as shown at 358 in FIG. 7. A volume scale may be provided forconfiguring the volume of the individual audio region relative to aglobal volume setting for that particular audio region type selected at346.

Various playback effects 414 for each audio region may be configured. Byway of example only, fade in and fade out settings may be selected asshown at 360 in FIG. 7. The fade in and fade out settings may be basedon time period as shown or alternatively based on index values, forexample word counts, or the like. The ramp or gradient or rate of thefade in or fade out may be linear or non-linear and configured asdesired.

The audio track selected for some audio regions may have a shorterduration than the duration of the audio region itself, depending on theeventual reading speed of the user, which dictates the eventual durationof each audio region. In such situations, it may be desirable toconfigure the audio track to seamlessly loop such that playbackcontinues for the entirety of the duration of the audio region. Looping416 may be activated or deactivated as shown at 362 in FIG. 7. Thelooping may apply seamless looping algorithms. The looping algorithmsmay be configured to minimise the sense of repetition of an audio trackor alternatively may repeat the audio track back-to-back if desired.

Real-Time Playback Effect Data

Optionally, some of the audio regions may further be defined byreal-time playback effect data that defines one or more controllableaudio playback effect settings that are to be varied across the audioregion or a portion of the audio region based on the user's real-timetext position. During playback of the soundtrack, the playback systemcontrols/varies the playback effects settings of the audio output systembased on the reading position counter (which is indicative of the user'sreal-time text position) for each audio region having pre-configuredplayback effect data. In situations where portions of the soundtrack hastwo or more overlapping audio regions, each having pre-configuredplayback effect data, the playback effect settings are variedindependently for each audio region on its respective separate audiochannel.

In this embodiment, the playback effect data is in the form of one ormore automation envelopes that are configured for at least some audioregions. An automation envelope defines one or more playback effects orfeatures or sound manipulations that are applied to the audio track ofan audio region in real-time during playback across the audio region ora portion of the audio region. For example, the automation envelope maydefine a controllable audio playback setting that varies in real-timethroughout the duration of the audio region playback, and which may bedependent on the real-time playback position relative to the playbackduration of the audio region.

In this embodiment, the automation envelope is a programmable functionconfigured to apply real-time sound manipulation to the audio playbackof the audio track based on the end user's reading position within theelectronic text. For example, the programmable function is dependent ona variable representing the user's real-time playback position (forexample the user's real-time text position corresponding to the word orcharacter they are reading) within the audio region playback durationsuch that the real-time sound manipulation applied to the audio trackplayback may be configured to vary as desired across the audio regionduration (which is dependent on the end user's reading speed). Theuser's playback position may be represented by the reading positioncounter corresponding to the index value of the word being read or ahigher or lower frequency counter being a product or function of thereading position counter if desired.

In one example, the programmable function may be configured such thatthe sound manipulation or playback effect setting that is varied duringplayback of the audio track across an audio region (or a portion of theaudio region) between the start index value and stop index value is afunction of the playback text position or text offset within or from thestart of the audio region, which may for example be defined based on theindex values of the audio region. The programmable function may beconfigured to provide a smoothed manipulation of the desired playbackeffect setting across the audio region to correspond to a smoothenvelope shape. In other embodiments, the automation envelope may bedefined by or stored in a look-up table that comprises a column listingthe playback positions in the audio region at the desired sampling rate(e.g. word or alternatively index value or alternatively a highersampling rate than each index value or other discrete positions withinthe audio region) and a column listing the desired playback effectsetting for each real-time playback position. The look-up table mayadditionally define transition properties between discrete positions,e.g. smooth transitions between changing degrees or levels ofapplication of a playback effect between discrete positions or any othertransition effect may be utilised.

The automation envelope may apply across the entire audio region or aportion of the audio region as discussed such that the playback effectsettings are manipulated or controlled in real-time during playbackaccording to the user's real-time text position in the electronic textat which they are reading (which has a corresponding position within theassociated audio region or regions). The automation envelope may haveany desired shape across the audio region, whether ramped, arbitrary,smooth, piece-wise, stepped, or any other linear or non-linear profilesor functions that vary with the playback duration. In one embodiment,the automation envelop shape may be defined by one or more discreteset-points configured for one or more index values representing textpositions (e.g. words) in the audio region and the overall remainingenvelope shape between the set-points may be interpolated to progress ortransition smoothly between the set-points or in a truncated manner orany other interpolation function may be employed.

In operation during playback, as will be explained in further detailwith respect to the soundtrack playback module, the frequency ofvariation or control of the playback effect setting match that dictatedby the automation envelope may be varied as desired to control theaccuracy of reproduction or application of the envelope shape to theaudio region. For example, in some embodiments the playback effectsetting may be manipulated or controlled as a rate or frequencycorresponding to the reading position counter rate, such that theplayback effect setting is modified to match the envelope on aword-by-word basis, i.e. is controlled or changed as required to matchthe envelope setting at each word in the audio region. Alternatively, ahigher frequency of control may be applied in which the playback effectsetting is modified at a frequency higher than that of the readingposition counter rate such that playback effect setting is controlled orupdated to match the envelope shape at least twice or multiple times foreach word in the audio region. In some embodiments, a separateautomation envelope update counter may be provided that has a frequency(typically higher) that is a product or factor of the frequency of thereading position counter or which is based on the user's reading speed.The soundtrack playback module may be configured to modify or update theplayback effect setting in real-time to match the automation envelopesetting at that playback position in the audio region defined by theautomation envelope update counter such that the playback effect settingis updated each time the automation envelope update counter increments,i.e. updating at the frequency of the automation envelope counter. Itwill be appreciated that the highest frequency of updating of theplayback effect setting to match the envelope is dictated by thehardware and/or sound processor used.

Two examples of automation envelopes will now be described, namelyvolume enveloping and panning enveloping, but it will be appreciatedthat the enveloping concept may be applied to any other playback audioproperty or effect setting that determines the nature or content of theaudio output, including but not limited to equalization, filtering,mixing, channel settings, reverberation, distortion, phase-shifting, orany other playback effect.

Volume enveloping or volume automation 418 may optionally be applied toeach audio region in parallel with or as an alternative to anyconfigured fade-in and fade-out playback effects, and as shown at 364 inFIG. 7. Referring to FIG. 9, activation of the volume envelope settinginitiates a GUI 370 for the volume envelope interface to enable the userto configure the desired volume envelope across the entire audio regionbetween the start text position and stop text position. Typically, thevolume envelope defines the playback volume setting across the audioregion relative to a default volume setting. The GUI 370 shows a grid ofvolume setting on the vertical axis between 0 to 100% and the indexvalues (or corresponding text from the electronic text) for the audioregion along the horizontal axis. In particular, a volume setting may bedefined across the audio region. For example, the volume setting isshown on the vertical axis of the volume envelope grid from 0-100% ofthe individual volume setting 358 previously described. Displayed alongthe horizontal axis of the volume envelope grid is the text region forthe audio region, including the words of the electronic text between thestart word and stop word for the audio region. In this embodiment, thevolume envelope 372 shape may be varied in a piece-wise fashion with avolume setting set-point being configured for each word or a selectionof words such that the overall envelope shape is formed or interpolatedbetween the discrete volume set-points as shown, although it will beappreciated that a higher resolution of volume settings may be providedin alternative embodiments if desired. The envelope shape betweenset-points may be any desired interpolation function, including asmoothing transition or direct point-to-point interpolation or any otherinterpolation method or function. In this embodiment, the volumeenvelope shape may be configured by moving the volume envelope line 372on the grid as desired. In FIG. 9 a relatively gradual increase involume from the start word at zero volume to 100% at approximatelytwo-thirds through the audio region is shown. Any desired volumeenveloping may be applied. It will also be appreciated that the volumeenveloping may be used as an alternative to applying fade-in andfade-out effects if desired or alternatively may be used in parallel.

Reverting to FIG. 5, a panning envelope 420 may optionally be applied toeach audio region, and as shown at 366 in FIG. 7. The panning envelopedefines a balance setting or weighting to be applied to the audio outputleft and right stereo signals for the left and right ears of a user. Inother words, the panning envelope defines the balance of the playbackvolume differential of the audio track between left and right stereosignals relative to a default balance setting across the audio regionbetween the start text position and stop text position. The defaultbalance setting may be centered or balanced with equal volume beingapplied to the left and right stereo channels. Referring to FIG. 10, thepanning envelope may be configured via a panning envelope interface GUI380, similar to the volume envelope interface 370. The GUI 380 shows agrid for a piece-wise panning envelope line 382 that is defined by abalance setting provided on the vertical axis with the audio regionrepresented by the index values (or the corresponding word equivalentsfrom the electronic text) across the horizontal axis between the startword and stop word as shown. In this embodiment, the balance settingvaries between +100% to −100%, with 100% representing full volume to theleft channel and nil volume to the right channel, and −100% representingfull volume to the right channel and nil to the left channel. Thebalance setting in this embodiment effectively defines a ratio of thevolume applied to the left and right channel. A setting of 0% equals100% volume applied to each left and right channel. A setting of +20%represents 100% volume to the left channel and 80% volume to the rightchannel. A setting of −50% represents 100% volume to the right channeland 50% volume to the left channel, and so on. Alternatively, it will beappreciated that any other form or scheme for balancing relative volumebetween the left and right stereo channels may be applied. Like for thevolume envelope, the panning envelope may be configured by dragging orshaping the panning envelope line 382 on the grid into the desiredshape. The balance setting may be configured on a piece-wise manner withthe balance setting set-points being determined for each word or aselection of words in the audio region, but alternatively a higherresolution of set-points may be applied if desired, and with theenvelope shape being determined by an interpolation function between theconfigured set-points. In the example panning envelope line 382 of FIG.10, the audio region starts with a centered balance and progresses intoa slightly left channel dominant balance setting, and then progressivelytracking toward a slightly right channel dominant balance setting, whichis then kept constant until the end of the audio region.

Configuring the panning envelope for audio regions enables variousplayback effects to be obtained, including providing a sense of movementof the sound or other static or dynamic spatial properties. It will beappreciated that the audio tracks of audio regions may have their ownstereo properties and that the panning envelope may be used in parallelto enhance those inherent stereo or spatial effects. Additionally, foraudio tracks without stereo effects or which are monaural, the panningenvelope may provide a sense of movement or spatial location to theaudio playback when heard by the user. The panning envelope may enhancethe reading experience, such that movement and/or the spatial propertiesof the audio may correspond to the content of the electronic text. Forexample, sound effects may be able to pan from the left ear to the rightear to create a sense of movement in the case of a text region whichrelates to a train passing by. Or if the electronic text refers to agunshot being heard to the left of a character, the appropriate balanceto the left channel may be applied to an audio region comprising anaudio tracking representing a ‘gun shot’ sound effect.

The volume and panning envelope settings may be applied to work incombination to produce the desired playback effects. Either of theconfiguration interfaces 370,380 may be activated to display both thevolume envelope line 372 and panning envelope line 382 on the same gridif desired.

Referring to FIG. 11, a GUI 390 is provided for configuring the globalvolume settings to be applied to audio regions having a particular audiotype. For example, audio regions having been defined with an ‘effect’audio type may have global volume settings comprising a minimum andmaximum volume and transformation configurations in which the volume ofindividual audio regions may be amplified or clipped to move theiroutput playback volume into the volume band or region defined by theglobal maximum and minimum volume settings.

Referring to FIG. 12, a GUI 395 is provided to enable a user to defineadditional audio types in addition to the default audio types of‘ambient’, ‘music’, and ‘effect’.

Referring to FIG. 13, the GUI 300 for the soundtrack creation system isshown for a portion of electronic text in which a number of audioregions have been configured for the soundtrack. The audio regions arerepresented graphically by the text underlines. In this embodiment,audio regions of a specific audio type are designated a specific colourof underline. As shown, audio regions may overlap as shown by theoverlapping underlines and the overlapping audio regions may be of thesame or different audio type.

Reverting to FIG. 5, as the audio regions are configured, the configuredaudio data for each audio region is stored in memory. FIG. 14 shows aGUI 397 showing a table of audio data for configured audio regions andtheir associated settings for a partially completed soundtrack. Eachline in the table represents a separate audio region. The columns definethe various audio playback properties configured for each audio region.

Reverting to FIG. 2, once the audio regions of the soundtrack have beendefined and configured, the audio data of the audio regions is compiled210 into an output data file 212 representing the soundtrack. The audiodata may be configured and compiled in various ways as will beappreciated, and the data may be compressed, encoded and/or encrypted.Essentially, the audio data is packaged or compiled into a data file orfiles in a manner that can be subsequently decompiled, extracted and/ordecoded by a user playback system that is configured to interpret thecontents of the soundtrack data file(s). It will be appreciated that thecollective audio data comprises the audio track playback properties foreach audio region described above, including the start word and stopword, and various other optional playback effects, including but notlimited to volume enveloping, panning enveloping, fade-in and fade-out,and the like. Additionally, the audio data for each audio region maycomprise the audio track sound file or alternatively links to the audiotrack file in an accessible database or music library for that audioregion may alternatively be used to reduce the soundtrack file size ifdesired. It will be appreciated that the compiling process may involvecompression and encryption in some embodiments.

The soundtrack output data file as shown at 212 is the output of thesoundtrack production or creation method. In addition to the audio data,the soundtrack data file may additionally incorporate a header file orheader data containing information about the sound file itself and theassociated corresponding electronic text to which it relates.Optionally, the soundtrack data file may incorporate the electronic textdata representing the electronic text itself such that the soundtrackfile is provided in the form of a multimedia data file comprising bothelectronic text (e.g. an e-book) for visual display and an associatedaudio soundtrack for playback.

3. A System and Method for Displaying Electronic Text and ControllingPlayback of a Soundtrack User Playback System—Overview

An embodiment of the user playback system for displaying electronic text(e.g. an e-book) and controlling playback of the associated soundtrackproduced by the soundtrack creation system above will now be describedby way of example with reference to FIGS. 15-24.

In this embodiment, the user playback system may be provided orimplemented as an application program or software running on anelectronic hardware system or platform that is the same or similar tothat described with reference to FIG. 1. Typically the user playbacksystem will be implemented on any suitable portable or non-portablehardware device or system having suitable processor and memorycapabilities, a display for displaying the electronic text, and an audiooutput system for playback of the soundtrack over speakers orheadphones. By way of example only, the hardware device or system may beportable consumer electronic devices such as smartphones (e.g. iPhone),cellphones, tablets (e.g. iPad), dedicated e-Reader or e-book devices,laptops, notebooks or any other suitable electronic devices.Alternatively, the user playback system may be implemented on a desktoppersonal computer or any other non-portable hardware system if desired.Typically, the user playback system is provided as an applicationprogram or software that may be loaded and run on the hardware devicesor systems mentioned above, but it will be appreciated that a dedicatedelectronic device may be configured and built as a stand-alone userplayback system if desired. It will be appreciated that the userplayback system may be implemented in the form of a website interface.For example, a user may access the website over the internet that isconfigured to provide a user playback system for viewing the electronictext and providing playback of the corresponding soundtrack via theuser's internet browser.

Referring to FIG. 15, the primary functions and data processing of anembodiment of the user playback system will now be described. Thevarious functions and data processing of the user playback system willbe described with reference to dedicated modules configured to carry outthe functions and data processing for clarity of description of the userplayback system. It will be appreciated that the modules may beconfigured or implemented in software in various ways and that theprocesses and functions performed by the modules may be integrated intoa reduced number of consolidated modules or further expanded andseparated into a larger number of modules depending on designrequirements.

In FIG. 15 the main modules of the user playback system 400 are shownand their interaction with various components of the hardware platformon which the user playback system is running, such as a display 110,user interface 108 and audio output system 112. The other aspects of thehardware system, such as processor 102, memory 104, data storage 106 andany other communication links or components are not shown for clarity.

The user playback system 400 comprises an electronic text display module402 that is configured to process electronic text data representing theelectronic text and controls display of the electronic text on thedisplay 110 in response to user input from the user interface 108. Thedisplay 110 may be a conventional electronic display (e.g. LCD or LEDdisplay screen) with the user interface 108 comprising any one or moretypical input devices such as, but not limited to, buttons, dials,switches, mouse, touch-pad, keyboard, voice commands or any othertactile, auditory or visual user input system or device. In a preferredembodiment, the hardware system is provided with an integrated display110 and user interface 108 in the form of a touch screen display, whichare commonly available on many portable electronic devices, such astablets, smartphones, notebooks and the like.

A reading position module 404 is provided for processing and variousreading position functions. For example, a reading position counter 406is configured to generate a reading position counter representing anestimated current reading position of the user in the electronic text. Areading position indicator module 408 is provided for displaying areading position indicator on the display representing the estimatedreading position of the user. A playback re-synchronization interfacemodule 410 is provided for enabling the user to re-synchronize theplayback of a soundtrack with their reading position or a new desiredreading position. A reading speed module 412 is provided for maintaininga reading speed variable representing the user's reading speed andfunctions for automatically or manually adjusting the reading speedvariable.

A soundtrack module 414 is provided for coordinating the playback of theaudio regions of the soundtrack over the audio output system 112 basedon the reading position counter to synchronize playback of thesoundtrack with the user's reading of the electronic text.

The user playback system is configured to receive and process thesoundtrack data file 212 and associated electronic text data file 204,whether combined into a multimedia data file or stored as separate datafiles, to generate the visual display of the electronic text and audioplayback of the accompanying soundtrack.

The individual modules of the user playback system 400 will now bedescribed in further detail below.

Electronic Text Display Module

The electronic text display module is configured to process and extractthe electronic text data from the e-book file 204. The electronic textdisplay module may be any text layout engine component for parsing andrendering electronic text for display on the display screen 110 as willbe appreciated by those skilled in e-Reader technology or electronictext display generally. One common example of such a text layout engineis UIWebView, which renders HTML content for display on a screen. Theelectronic text display module provides the user with various navigationtools or a navigation interface that is operable to traverse the pagesor content of the e-book as desired, including page-skip, page-turn,chapter-skip, scrolling and the like, as will be appreciated by thoseskilled in e-Reader technology. The navigation tools or interface may beoperated via the user interface 108, such as via interaction with atouch-screen display, as is known in e-Reader technology.

Referring to FIG. 21, the electronic text displaying module may beprovided with a font setting interface 540 to enable the font size ofthe display or other features of the display to be altered as will beappreciated by those skilled in e-Reader technology.

Reading Position Counter

The reading position counter 406 generated by the reading positionmodule 404 represents the estimated current text position in theelectronic text that the user is currently reading. In this embodiment,the counter may represent the index value corresponding to the user'sestimated current text position. In this embodiment, the readingposition counter increments at a rate based at least partly on aconfigurable user reading speed variable, which may be generated andmaintained by the reading speed module 412 to be explained later.

In this embodiment where the electronic text has been indexed based onnumber of words, the reading position counter may represent the wordnumber or word count corresponding to the estimated position (word) inthe electronic text that the user is reading. The counter is configuredto increment at a rate based on the reading speed variable, which may beprovided as a numerical value such as words per minute (wpm) or anyother suitable reading speed metric. By way of example, if the readingspeed variable is set to 400 wpm, then the reading position counter willincrement by 1 every 0.15 seconds.

The reading position counter may be modified or controlled by datareceived from the electronic text display module representing the user'sinteraction with the electronic text (i.e. which page they are readingor page skipping). In some embodiments, the reading position counter maybe configured such that the initial reading position counter valueautomatically corresponds to the index value of the first word on anynew page selected for reading or the page on which reading commencesbased on data received from the electronic text display module. Thecounter may then increment at a rate based on the reading speedvariable. In some embodiments, when the end of a page is reached by thereading position counter, a ‘page turn’ or ‘new page’ transition mayautomatically occur such that the counter increments continuously andprovides for automatic page turning. In other embodiments, the readingposition counter may be configured to pause or halt at the last word ofa page, awaiting to commence again based on a user ‘page turn’ or ‘nextpage’ input.

A ‘reading pause’ button or input may be provided for the user toactivate when they wish to pause reading and playback of the soundtrackmodule, and the reading position counter may halt when a ‘reading pause’is initiated, which will also cause a pause in the playback of thesoundtrack.

Reading Position Indicator Module

The reading speed position indicator module 408 is configured togenerate a visible moving reading position indicator on the display 110representing the current estimated text position in the electronic textthat the user is reading. The reading position indicator moduleinteracts with the electronic text display module to present a readingposition indicator on the display based on the reading position counter.

In one embodiment, the reading position indicator may be an ‘inline’indicator that is displayed within the lines of text. For example, inthis word-based indexing embodiment the reading position indicator maybe configured to traverse the lines of text word-by-word such that theposition of the indicator corresponds to the word represented by thecurrent reading position counter so as to highlight, identify orindicate the current estimated word being read. The inline indicator maybe any form of a marker, including an underline as shown at 510 in FIG.16, a marker dot or ball 512 located in the vicinity or adjacent (e.g.above, below or adjacent on sides) of the word being read as shown inFIG. 17, or any other marker or system for highlighting the wordcorresponding to the real-time reading position counter value, includingbut not limited to a change in font size, font shape, or font colour ofthe word corresponding to the reading position counter value, or achange in text background colour in the vicinity or associated with theword. The marker may jump from word-to-word, or traverse continuouslybetween the words.

In other embodiments, the reading position indicator may be a ‘margin’or ‘line’ indicator that is displayed in the page margin (left or right)of the electronic text, and which is configured to slide, track or movedown the page line-by-line such that the position of the indicatorcorresponds to the line of electronic text containing the currentestimated word being read based on the current reading position countervalue. The ‘margin’ indicator may be configured to jump from line toline, or alternatively continuously move down the margin such that itsposition reflects the estimated word position within each line currentlybeing read. For example, a slider at the top of a new line may representthe reading position counter corresponds to the user reading the startor first word of the line, a slider in the middle of the linecorresponds to the middle portion or words of the line, and a sliderbelow the line represents the reader is at the end of the line andtransitioning into the start of a new line. An example of such a‘margin’ indicator is shown at 514 in FIG. 18. In other embodiments, the‘line’ indicator may be an underline, or highlight, or change in fontsize, or change in font shape, or change in font colour, or a change intext background colour of the entire current line of text being read.

As shown in FIG. 19, a tools menu 516 may be provided to enable a userto determine or select which reading indicator(s) to display at 518,including the indicators, such as the ‘underline’ or ‘ball’, and the‘margin’ indicator, such as the ‘slider’. It will be appreciated thatthe user may activate both an and ‘margin’ indicators to be displayedtogether. Alternatively, the reading position indicators may bedeactivated if the user desires.

Playback Re-Synchronization Interface Module

A playback re-synchronization interface module 410 is provided in thisembodiment to enable a user to re-synchronize playback of the soundtrackwith the user's current reading text position should synchronization belost for any reason such as reading interruptions or reading pauses, orif the user wishes to commence reading at a new particular arbitrarytext position in the electronic text, whether behind or forward of thecurrent reading position.

The playback re-synchronization interface module is configured torespond to particular user re-sync input that indicatesre-synchronisation is required. The user re-sync input comprisesinformation indicative of the desired re-sync text position in theelectronic text where the reader is currently reading or wishes tocommence reading.

In one example, the user re-sync input may be in the form of a userhighlighting or indicating the re-sync text position via tactile inputinto the touch screen display. For example, the user may tap,double-tap, or press on the word or position in the electronic textcorresponding to the desired re-sync text position. Alternatively, theuser may provide a re-sync input via moving a cursor or pointer on thedisplay via a user input device such as a PC mouse or touch-pad orsimilar, and selecting the word with a mouse click or operation ofanother associated button of the user input device. In response to thisuser re-sync input, the re-synchronisation interface module isconfigured to reset or modify the reading position counter 406 to matchthe index value corresponding to the word at the re-sync text position.As will be explained in further detail later, the playback of thesoundtrack is controlled by the soundtrack playback module 414 and anymodification of the reading position counter will be sensed by thesoundtrack playback module 414, which responds by halting currentsoundtrack playback and re-starting or re-cueing the soundtrack playbackto correspond to the new reading position counter. As will be explainedin further detail later in regard to the soundtrack playbackcommencement, the soundtrack playback module is configured tointelligently cue the entire soundtrack corresponding to the new readingposition counter. This involves initiating playback of any one or moreactive audio regions at the new counter position, including cueing theplayback positions at offset cue-times within the audio tracks of theaudio regions to correspond to the position of the new counter withinany active audio regions. For example, the new counter position may notbe at the start of an audio region, but could be part way through anaudio region and in such cases the audio track is cued to initiateplayback at an offset cue-time part-way through the audio trackaccordingly, rather than at the default start time=0s of the audiotrack. In this embodiment, the re-synchronisation module operates in thebackground and monitors and responds to a user re-sync input by theuser.

In another example, the user re-sync input may be in the form of a userdragging the reading position indicator (‘inline’ or ‘margin’) to thenew desired re-sync text position via interaction with the touch screendisplay, or via operation of a user input device. Once the readingposition indicator is moved to the re-sync text position, there-synchronisation interface module is configured to reset or modify thereading position counter 406 to match the index value corresponding tothe word at the re-sync text position as in the above example, whichthen causes the soundtrack playback to be re-synchronised to correspondwith the new position.

Reading Speed Module

The reading speed module 412 maintains a reading speed variablerepresenting the user's estimated reading speed. In this embodiment, thereading speed variable is a numerical value corresponding to words perminute (wpm) although any other metric for representing or measuringreading speed may alternatively be used.

Initial Setting

When a user begins reading a new e-book, the reading speed variable maybe automatically configured to have a default value representing theaverage reading speed for a population or alternatively the readingspeed module may be configured to access stored historical reading speeddata for the user and load the user's historical reading speed into thereading speed variable initially. This initial setting may then beadjusted or customised manually and/or automatically to the user beforeor during reading using any of the techniques described below.

Reading Speed Test

Referring to FIG. 19, a reading speed test may be activated 524 in thetools menu 516 to initiate a reading speed test interface. The readingspeed test calculates a reading speed value (e.g. wpm) based on the timerequired for the user to read a default portion of text, the length ofwhich is known. The calculated reading speed value is loaded into thereading speed variable of the system to replace the initial or currentsetting.

Manual Adjustment

The reading speed variable may be manually adjusted, either directly orindirectly, by the user at any stage of the reading process as desired,whether in real-time during reading of the electronic text and playbackof the soundtrack, or before reading has commenced or during a readingpause, when the soundtrack playback is paused. Any one or more of thefollowing methods may be offered to the user to adjust their readingspeed variable.

By way of example, some direct manual adjustment methods follow.

In one example, referring to FIG. 19, a slide scale 522 may be providedwhich the user may operate to increase or decrease their reading speedvariable in the tools menu 516.

In another example, a reading speed input field may be provided for theuser to manually input their numerical reading speed.

In a further example, the may be presented with a user adjustmentinterface in the form of ‘increase’ or ‘decrease’ adjustment buttons(e.g. +/− buttons) to incrementally increase or decrease the currentreading speed variable by predetermined value in a real-time readingspeed adjustment interface during reading. The adjustment buttons may bepresented to the user on the display alongside or in an area adjacent tothe electronic text and when activated, the reading speed module mayactivate an ‘inline’ or ‘margin’ reading position indicator to appear(if not activated already) such that the reading position indicatorsmoves relative to the text (either inline or in the margin as previouslydescribed) at a rate dependent on the reading position counter, which isin turn dependent on the reading speed variable. The user may thenadjust the reading speed variable using the adjustment buttons until thereading position indicator traverses the text at substantially the samespeed or rate as the user reads the text, i.e. such that the readingposition indicator follows the current word being read. The readingposition indicator provides the user with visual feedback as to thecurrent reading speed variable setting and assists in the calibration ofthat variable to match the user's reading speed during real-time readingof the electronic text. In particular, as the electronic text andreading position indicator are simultaneously displayed with the useradjustment interface on the display screen, the user can view inreal-time the effect of any adjustment of the reading speed variablebecause the rate of movement of the reading position indicator changesaccording to their adjustment. Once adjustment is complete, the readingposition indicator may disappear after a predetermined period ofinactivity in adjustment should the reading position indicator(s) havebeen deactivated by the user previously. In alternative embodiments, theuser adjustment interface may be in the form of physical +/− buttons, oran operable dial (graphical on the display or a physical dial) which isconfigured to increase the reading speed variable when rotated in onedirection, and decrease the reading speed variable when rotated in theother direction, or an operable slider (graphical on the display or aphysical slider) that is movable along a scale and which is configuredto increase the reading speed when slid to one end of the scale, anddecrease the reading speed variable when slid to the other end of thescale, or any other suitable user adjustment input for increasing ordecreasing the reading speed variable.

By way of example, some indirect manual adjustment methods follow.

In one example, a reading speed adjustment may be provided in the formof a tracking interface. When initiated, the tracking interface isconfigured to sense the rate or speed of movement of a user's finger (orinput device representing the user's finger such as a stylus) on thesurface of the touch screen or movement of a cursor or pointer on thedisplay screen controlled by a user input device such as a PC mouse ortouch-pad or similar, either in-line with the text from left to right orright to left word-by-word (e.g. a substantially horizontal movementacross the displayed text), or down or up the page of text line-by-line(e.g. a substantially vertical movement relative to the displayed text),for a tracking period, depending on the reading direction of thelanguage of the electronic text. The user moves their finger or otherinput to trace the words or line of the electronic text on the displayas they read them. In this embodiment, the speed or rate of movement maybe calculated based on the number of words traversed and the time takenfor a tracking period. The speed or rate of movement calculated ormeasured, for example in wpm, can then be used to update the readingspeed variable. The start and end of the tracking period may be definedbetween when the user initiates and ceases the movement. In the fingercontact embodiment, the movement is initiated when the user touches thescreen with their finger to initiate a movement and ceases when the userdisengages their finger from the touch screen for example. In thepointer or cursor embodiment, the movement is initiated when the useractuates a control button associated with the user input device, e.g. amouse click or similar. In this reading speed adjustment, the userfollows the text they are reading with their finger on the screen,whether ‘inline’ with the text word-by-word horizontally or down thepage of the text line-by-line vertically, whether in the margin adjacentto the text or within the text. In one form, the quantity of text reador traversed is calculated by the text (e.g. number of words orcharacters) located between a start text position of the movement andthe end text position of the movement (e.g. based on the differencebetween the index values at the detected start and stop text positions).In another form, the quantity of text read during the movement isdetermined by incrementing a counter after each new word or character ispassed by the finger, input device, or cursor or pointer, when inlinetracking is employed. In this form, the measurement of reading speed maybe periodically or continuously updated during the movement based on thecounter and time taken from the start of the movement.

In another example, similar to the previous tracking interface example,a reading speed adjustment may be provided in the form of a slider draginterface. This operates in a similar manner to the tracking interface,except that the user is presented with a reading position indicator(‘inline’ or ‘margin’) in the form of a slider which they drag or slidevia an operable user input (whether a finger in the context of a touchscreen display, or a cursor or pointer controlled by a PC mouse ortouch-pad or similar) along the lines of text (‘inline’) or down themargin of the text (‘margin’) to reflect their reading position as theyread. In particular, the user input should drag the reading positionindicator to follow the text they are reading as they read it. Theslider drag interface is configured to calculate or measure the rate orspeed of the traverse of the slider based on the number of wordstraversed and the time taken for a tracking period, and the same optionsas described for the previous tracking interface example also apply tothe slider drag interface. As with the previous example, the calculatedor measured speed or rate of movement, for example in wpm, is then usedto update the reading speed variable. The slider may be configured orrestricted to particular movement paths in-line with the text,traversing word-by-word (or character-by-character), line-by-line, or ina path along the margin of the page of electronic text if desired toguide the traverse for the measurement.

Automatic Adjustment

Additionally or as an alternative to the user manual adjustmenttechniques above, the reading speed module may be provided with anautomatic adjustment module that is configured to automatically adjustor update the reading speed variable based on the rate at which the useris traversing the electronic text as determined by arbitrary user inputsindicative of their current reading position within the electronic text.As shown in the tools menu 516 in FIG. 19, the reading speed module maybe provided with an interface 520 for activating or deactivating theautomatic adjustment module. An embodiment of this automatic adjustmentmethod will be described with reference to FIGS. 22-24.

With reference to FIG. 22, the automatic adjustment module is configuredto continuously sense 602 or detect a specific ‘reading position userinput’ that is indicative of the current text position (e.g. word orcharacter) the user is reading. The reading position user input may be apre-configured or pre-determined type of user input that is monitored bythe automatic adjustment module. If the user playback system comprises atouch screen display, the reading position user input may be in the formof a ‘touch input’ such as a user tapping or pressing on the surface ofthe display screen at the location of the word or character they arecurrently reading, whether with their finger or a user input device,such as a stylus. The tapping may be a single tap or a double-tap, andthe automatic adjustment module is configured to sense for such aninput. Alternatively, the reading position user input may be in the formof a display cursor or pointer being moved by the user via a user inputdevice such as a PC mouse or touch pad or tracker ball or similar, tothe current word or character they are reading and selecting that wordor character with an action button of the user input device, for exampleby clicking the mouse button or other button associated with the type ofuser input device employed. The reading position user input may be anyarbitrarily selected text position within the electronic text thatcorresponds to the word or character the user is currently reading, andmay occur at any arbitrary time determined by the user. The user maydetermine the frequency of the reading position user inputs, and thetime interval between inputs may be random. For each page of electronictext displayed, the user may arbitrarily make none, one, or multiplereading position user inputs as they read the electronic text on thepage. For example, as the user is reading a page on a touch screen theymay tap or press on any word or otherwise select or indicate the word asthey read it, and the automatic adjustment module will detect this as anew reading position user input. Each time a new reading position userinput is detected, the index value (e.g. word or character count)corresponding to the user selected text position (word or character) inthe electronic text is stored 604 in memory, along with a time stamp ofwhen the input was detected.

Upon detection of a new reading position user input, the automaticadjustment module is configured to calculate a new estimate of readingspeed 606. Referring to FIG. 23, the reading speed calculation will beexplained. Firstly, the index value and time stamp of the last storedreading position user input is retrieved 620 from memory. Then, theindex value and time stamp corresponding to the new reading positionuser input is retrieved 622 from memory. The quantity of text (e.g.number or words or characters) between the last and new readingpositions is then calculated 624 by virtue of the difference between therespective index values. The time period between the last and newreading positions is then calculated 626 by virtue of the differencebetween the respective time stamps. Finally, a new estimate of readingspeed (e.g. in wpm or wps) is calculated 628 based on the length of textand time period determined. It will be appreciated that in alternativeembodiments time stamps need not be determined, and that a separatetimer may be utilised to record the time period between successivereading position user inputs.

Reverting to FIG. 22, once the new estimate of reading speed has beencalculated, an update algorithm is initiated to generate an updated userreading speed variable. In some embodiments, the update algorithm maysimply replace the existing reading speed variable with the new estimateof reading speed. In other embodiments, the update algorithm may apply amoving or weighted average to a number of the most recent calculatedestimates of reading speed. Additionally, the update algorithm mayemploy a probability function to the new estimate of reading speed tominimise the weight applied to anomaly or erroneous new reading speedestimates that are outliers relative to the current reading speedvariable, for example caused by a reading interruption or a userprematurely page-skipping or similar.

With reference to FIG. 24, one example of an update algorithm 608 willbe described by way of example only. The update algorithm retrieves 630the new estimate of reading speed (ES). A new moving average (MS) of thereading speed variable is then calculated 632 based on a predeterminednumber of recent reading speed variables. The moving average may beweighted or non-weighted. In this embodiment, the moving average isweighted toward the most recent reading speed variable. By way ofexample only, the moving average may be calculated asMS=(NS₁×0.4)+(NS₂×0.3)+(NS₃×0.2)+(NS₄×0.1), where NS₁ is the most recentreading speed variable, and NS₄ is the fourth most recent. It will beappreciated that a non-weighted moving average may alternatively beemployed if desired. A difference value indicative of the percentagedifference or change (DS) between the new estimate of reading speed ESand the moving average MS is then calculated 634 as DS=|(MS−ES)/MS|. Aprobability weighting for the new estimate of reading speed is thencalculated 636 based on a continuous probability distribution. In thisexample, a normal or Gaussian distribution is employed, although otherdistributions may alternatively be utilised. The probability weighting(GV) is calculated based on the following probability density function:

${G\; V} = {{f( {D\; S} )} = {^{- \frac{{DS}^{2}}{2\sigma^{2}}}/\sqrt{2{\pi\sigma}^{2}}}}$

where σ² is the width of the distribution, and may be selected asdesired to suit typical reading speed variations. Finally, the newreading speed variable (NS) is calculated 638 based on the new estimatedreading speed (ES), the moving average (MS), and the probabilityweighting as: NS=MS+(ES−MS)×GV.

Reverting to FIG. 22, once the update algorithm processing is complete,the automatic adjustment module is configured to update the user readingspeed variable to the value of the new reading speed variable NS. Theautomatic adjustment module then returns to the initial step 602 ofmonitoring for a new reading position user input.

In some embodiments, the user reading position input may be configuredto correspond to the re-sync input of the playback re-synchronizationinterface module such that when the user initiates a user readingposition input, both the automatic adjustment of the reading speedvariable is activated along with the re-synchronisation of thesoundtrack with the user selected text position (corresponding to there-sync text position). Alternatively, the user reading position inputand re-sync inputs may be configured to be different types of inputssuch that the automatic adjustment module and re-synchronisationinterface module can be triggered or activated independently.

Soundtrack Playback Module

The soundtrack playback module 414 is configured to process the audiodata from the soundtrack data file 212 and coordinate real-time playbackof the audio regions of the soundtrack based on the reading positioncounter 406 to synchronize playback of the audio regions of thesoundtrack with the user's reading of the electronic text.

Overview

The soundtrack playback module 414 controls playback over the audiooutput system 112, which may have multiple channels that can playbackconcurrently or which may be mixed together into an output audio signal.In some embodiments, the audio output system may be configured to simplyreceive, amplify, and output an audio output signal from the soundtrackplayback module as audio sound output over connected stereo speakers,headphones, earphones or the like. In such embodiments, the soundtrackplayback module is configured to directly load and process the audiotrack files (digital) from the soundtrack data file or from a linkeddatabase or music library, and apply the required audio processing andany playback effects to generate the analogue audio output signal forplayback over the audio output system. In other embodiments, the audiooutput system may be a complete audio controller with sound processor(e.g. DSP) that is configured to load and process the audio track filesfor the audio regions and generate the required analogue audio outputsignal for playback over connected speakers, headphones, earphones, orsimilar, in response to control signals from the soundtrack playbackmodule. In such embodiments, the soundtrack playback module isconfigured to generate the control signals for controlling the operationof the audio output system to load and playback the audio tracks of theaudio regions in a coordinated manner with the required playbackeffects. The control signals may include, by way of example only and notlimited to, control signals for loading audio tracks (e.g. audio files)for playback, adjusting or allocating playback channels, adjusting orcontrolling equalizer settings, and/or adjusting volume and balancesettings in accordance with fade-hi, fade-out, or panning envelopeconfigurations and volume envelope configurations as referred to belowfor example.

Application of Real-Time Playback Effect Data

As mentioned above, the soundtrack playback module is configured tocontrol in real-time the playback effect settings of the audio outputsystem according to any pre-configured playback effect data (e.g.automation envelopes) of an audio region, or this may be controlled byan automation envelope sub-module. If two or more audio regions areplaying concurrently on different channels, the soundtrack playbackmodule is configured to independently vary the playback effect settingsassociated with each channel according to any pre-configured playbackeffect data defining the audio region playing on each channel. Forexample, a panning envelope may be applied to a sound effect audioregion playing on one channel while a volume envelope is simultaneouslybeing applied to a background music audio region on another channel.

By way of further explanation, the audio data for an audio region maycomprise one or more automation envelopes that define one or morerespective playback effect settings to be varied across the playback ofthe audio region in real-time based on the user's real-time textposition (which corresponds to a playback position relative to the audioregion playback duration). The soundtrack playback module is configuredto vary one or more playback effect settings of the audio output systemor perform sound manipulation of the audio track of an audio regionaccording to the automation envelope(s) defined in the audio data ofthat region during the playback duration of the audio region. In oneembodiment, the playback effect settings may be modified or updated tomatch the automation envelope at a frequency corresponding to thereading position counter or the playback effect setting may be updatedto match the automation envelope at a higher frequency than the readingposition counter. For example, a separate automation envelope updatecounter may be provided that increments at a frequency higher than thatof the reading position counter, and may or may not be based on theuser's reading speed variable. As previously described, the automationenvelope for an audio region may be stored in the audio data as aprogrammable function that is dependent on the real-time playbackposition within the audio region, or a look-up table, by way of exampleonly.

By way of example, the soundtrack playback module may be configured forreal-time volume adjustment according to a volume envelope, or real-timebalance adjustment according to a panning envelope. The soundtrackplayback module may be configured to directly modify the audio trackplayback properties with audio or sound pre-processing of the audiotrack and/or via controlling the settings and controls (volume, balance,equalizer etc) of the audio output system via control signals.

Soundtrack Playback Commencement

Playback of the soundtrack may be commenced at any text position withinthe electronic text, e.g. e-book. The commencement text positions may beautomatically selected based on the user's interaction with the e-bookand/or selected manually by the user when desired. For example, the userplayback system may provide a ‘play’ or initialization or commencementinput or trigger, e.g. play button or the like, that is operable toinitiate soundtrack playback based on a commencement text position orplayback may be initiated automatically as the user opens the book orturns to a new page. In this embodiment, absent any user manualselection of the commencement text position, the commencement textposition automatically defaults to the first word of the current pagenavigated to by the user. Alternatively, the user may manually select adesired arbitrary commencement text position, such as by using there-synchronisation input module and providing a re-sync input toindicate a re-sync text position (equivalent in this context to acommencement text position). The soundtrack playback module isconfigured such that playback of the soundtrack may be commenced at anyarbitrary text position within the electronic text, whether that is atthe start, end, or part-way through one or more audio regions, or atpositions where there are no active audio regions.

The commencement text position corresponds to an index value in theelectronic text, e.g. a particular word in the electronic text. Thereading position counter is updated to match the commencement text valueof the commencement text position once selected. When playback of thesoundtrack is initiated for a particular commencement text position, thesoundtrack playback module is configured to assess the audio data of thesoundtrack for any audio regions that are active in the soundtrack atthat commencement text position, i.e. any audio region durations(defined between their respective start index value and stop indexvalue) that encapsulate the commencement index value corresponding tothe commencement text position. If there are no active audio regions orif the commencement text position only corresponds to the start or endtext positions of one or more audio regions, then the soundtrackplayback module is configured to commence playback and cue the audioregions of the soundtrack for playback in a ‘normal playback mode’(discussed further below) as the audio regions to be cued will allinitiate their associated audio tracks at the default start of the audiotrack (e.g. time=0s). If the commencement text position occurs withinone or more audio regions, then the soundtrack playback module isconfigured to commence playback and cue the audio regions of thesoundtrack playback module in a ‘initialisation playback mode’(discussed further below) as the audio regions to be cued will need tostart their associated audio tracks at cue times part-way through thetrack, rather than at the default start time (e.g. time=0s) of the audiotrack, to correspond to the commencement text position occurringpart-way through the audio region.

An example of the ‘initialisation playback mode’ will be described infurther detail below. The objective of the initialisation playback modeis to cue the audio tracks of the audio regions to begin at offset cuetimes part-way into the audio track based on the user's reading speed sothat the soundtrack can be accurately commenced at any text position.

The process 600 of calculating the audio track cue time is carried outfor each audio region in which the commencement text position occurswithin its audio region duration, i.e. part-way through the audioregion. By way of example, the process for calculating a cue time willnow be explained for a scenario in which the audio region has a durationdefined by a start index value of S1=200 (e.g. 200^(th) word) and thestop index value is S2=600 (e.g. 600^(th) word), the commencement indexvalue (corresponding to the commencement text position) is C=300 (e.g.300^(th) word), and where the reading speed variable is RSV=300 wpm (5words per second, wps). Firstly, the commencement offset value OV forthe commencement index value within the audio region is determined asthe difference between the commencement text value and the start indexvalue of the audio region, OV=C−S1, 300−200=100 words, i.e. thecommencement text position occurs 100 words from the start of the audioregion. Then the cue time CT for the audio track of the audio region iscalculated based on the commencement offset value and the reading speedvariable, e.g. CT=OV/RSV, 100/300=1/3 minute=20 seconds. In thisscenario, the audio track for the audio region is set to commence orstart at an offset cue time of 20 seconds into the track, rather than atthe default beginning of the audio track (e.g. time=0s). In other words,the offset cue time for the audio track of the audio region correspondsto the offset cue time within the audio track relative to the defaultstart time (e.g. time=0s of the audio track) and this is calculatedbased on the offset position of the commencement text position from thestart of the audio region and the user's reading speed.

Once all the cue times for all active audio regions at the commencementtext position are calculated in the initialisation playback mode, thesoundtrack playback module is configured to simultaneously triggerplayback of the audio tracks of the one or more active audio regions tobegin at their respective cue times when playback is activated orinitiated. In some cases, one or more of the offset cue times of theaudio regions may coincidentally correspond to a position having a peakin the volume of playback, for example as dictated by a volume envelopeassociated with the audio region or due to the natural volume variationof an audio track. To avoid the user experiencing a sharp and abruptsoundtrack initiation at a loud volume, the initialisation playback modemay be optionally configured to apply a gradual volume fade-in over apreset or configurable time period during commencement of the soundtrackat the commencement text position. After commencement of the soundtrack,and optional fade-in, the soundtrack module is then configured to revertto the ‘normal playback mode’ for the remainder of the soundtrack. The‘normal playback mode’ is discussed further below.

Soundtrack Playback Real-Time Cueing (Normal Playback Mode)

In this embodiment, the soundtrack playback module 414 cues playback ofthe audio regions in real-time based on the reading position counter.For example, playback of an audio region is initiated when the readingposition counter matches the start index value of the audio data forthat audio region. Likewise, playback of an audio region is stopped whenthe reading position counter matches the stop index value of the audiodata for that audio region.

The soundtrack playback module continuously or periodically monitors thereading position counter to coordinate and cue playback of the audioregions. As will be appreciated, the soundtrack playback module 414 maycoordinate playback of concurrent or overlapping audio regions onmultiple different channels. The soundtrack playback module is alsoconfigured to modify the playback properties of the audio regions toreflect any playback effect configuration settings of the audio data,such as but not limited to, fade-in and fade-out effects, balancesettings in accordance with any panning envelope, and volume envelopesettings as set out above. For example, the playback effects of an audiotrack of an audio region may be modified throughout the audio region tocomply with any playback effects that are configured based on thereading position counter. For example, the playback effects may remainconstant or vary throughout an audio region depending on the configuredaudio data. In this embodiment, the reading position counter representsthe current word being read and the soundtrack playback module canmodify the playback effects of the audio regions to match the user'scurrent text position in real-time in accordance with the pre-configuredaudio data.

The soundtrack playback module coordinates the playback of the audioregions by continuously and/or periodically searching and analyzing theaudio data, for example at each new reading position counter incrementor some higher frequency, to assess whether any new audio regions shouldcommence playback, whether playback of any audio regions should cease,or whether the playback effects or audio properties (e.g. volume orbalance) should be modified for currently playing audio regions. Basedon this analysis, the soundtrack playback module alters and cuesplayback of the audio regions to accord with the audio data with thetiming for synchronisation with the user's reading being provided by thereading position counter.

Looping or Truncation Control

It will be appreciated that the eventual or actual ‘playback duration’(time between the start index value and stop index value) of an audioregion of the soundtrack is dependent on the user's reading speedvariable. For some audio regions, the eventual ‘playback duration’ maybe longer than the natural duration of the audio track of that audioregion, and this may occur more often for slower readers. When loopingis not configured, silence will occur in relation to that audio regionbetween the end of the audio track and the end of the audio region (whenthe stop text position is reached). When playback is configured to loop,the audio track will seamlessly loop until the end of the audio region.For other audio regions, the ‘playback duration’ of the audio region maybe shorter than the natural duration of the audio track for that region,and this may occur more often for faster readers. In such situations,the soundtrack playback module halts or truncates the playback of thenatural duration of the audio track when the end of the audio region isreached, or may do this in combination with fade-out effects ifconfigured.

As previously mentioned, a ‘reading pause’ button or input for the usermay be provided that can be activated to halt the counter fromincrementing. The soundtrack playback module is configured to senseactivation of a ‘reading pause’ input and responds by halting audioplayback until the ‘reading pause’ is deactivated such that the counteris incrementing again, at which point the soundtrack continues playingfrom where it was paused.

The soundtrack playback module may be provided with an interface forconfiguring audio or playback settings. For example, FIG. 20 shows anaudio settings interface 530 which provides global volume settingsliders 534 for configuring the global volumes for the playback of audioregions having different audio types, such as ‘ambient’, ‘sound effects’and ‘music’. This enables a user to selectively and individually modifythe playback volume of audio regions having particular audio types.Additionally, the user has the ability to activate or deactivate (i.e.mute) the audio regions for a particular audio type. For example, if theuser prefers not to hear ambient audio regions, they may deactivatethose sounds using the appropriate on/off buttons shown at 532.

4. Other Alternative Embodiments and Features Text Indexing

In the embodiment described above, the soundtrack creation and playbacksystem is based on text indexing of the electronic text based on thenumber of words or characters. For clarity, the above embodiments focuson a word-based indexing approach. However, it will be appreciated thatthe system may be modified to work on a higher-resolutioncharacter-based indexing approach or other text position referencing.For example, the electronic text may be indexed based on the number ofindividual characters in the electronic text, including letters,symbols, punctuation and the like. For example, each character may bedesignated an index value representing its character number or charactercount in the electronic text. Under this indexing scheme, the start andstop index values defining a text region for an audio region correspondto a start character and stop character in the electronic text for thetext region. All modules, functions and processes above may be modifiedto work on a character-based indexing scheme. For example, the readingposition counter would represent the index value corresponding to thecurrent estimated character being read by the user. This approach may beuseful particularly for non-English symbol based languages as referredto below.

Different Languages

It will be appreciated that the soundtrack creation and playback systemmay be adapted to suit any language or written text format. Theembodiments described above are in relation to English text written andread horizontally from left to right in rows, with rows being orderedtop to bottom of the page. However, it will be appreciated that thesystem may be modified to suit non-English languages, including Japanesetext which is written and read in columns from top to bottom, and withthe columns ordered left to right on the page. Modification of thesystem to suit such languages may require the orientation and traversepath of reading position indicators to be modified. Additionally, acharacter-indexing approach to the text may be required or preferred forsome languages.

Adaption of Existing E-Readers

It will be appreciated that the user playback system described above maybe adapted to augment existing e-Reader software, devices or systems. Adedicated or integrated user playback system providing both text displayand soundtrack playback back is not essential. It will be appreciatedthat the soundtrack playback module and reading position module may runas a separate application program and communicate or interact withexisting or third-party e-Reader systems or application programs, forexample via an API, to exchange relevant data to provide the readingexperience of text display and a synchronised soundtrack playback above.For example, third-party e-Reader systems may control the text displaybased on the imported or loaded e-book data file, while the soundtrackplayback module and reading position module may coordinate thesoundtrack playback with the user's reading speed based on thecorresponding soundtrack data file.

Application of Soundtrack to a Portion of Electronic Text

The embodiments above have been described in the context of a soundtrackbeing configured for an entire electronic text. However, it will beappreciated that the soundtrack may be applied to select parts orportions within an electronic text, and need not be applied to an entireelectronic text.

5. Advantages and Benefits

The following advantages may apply to at least some of the embodimentsof the system and method described.

The soundtrack creation system and method, and user playback system andmethod, provide for increased accuracy in synchronising soundtrackplayback with the reading speed of a user for electronic text. Thesystem provides for multiple channels of audio to enable concurrentplayback of two or more partially or entirely overlapping audio regionssuch to create an audio output comprising, for example, sound effects,ambience, music or other audio features that are triggered to playbackat specific portions in the electronic text to enhance the readingexperience. The text indexing approach to configuring the audio regionsand triggering their playback with the user's reading speed provides aflexible and robust approach to synchronisation of the audio with theuser's reading speed, and the reproduction of desired playback effects.For example, the synchronisation is not affected by font-size changes orpage layout formatting, settings which are often modified or customisedby the end user when reading an e-book.

The system also provides various unique and user intuitive manual andautomatic adjustment options for configuring and/or adjusting and/orcalculating the reading speed variable to match the user's readingspeed. Additionally, the system enables a reading position indicator tobe displayed that is customised to traverse at a rate matched to the enduser reading speed.

The system enables seamless soundtrack initiation at any text positionin the electronic text, from automatically selected text positions touser-selected arbitrary text positions with offset cue times for audiotracks being automatically calculated and applied for any active audioregions at the commencement text position.

The system allows for automation envelopes of various playback effectsto be configured and applied across audio regions, or portions of audioregions, and the application of the envelopes occurs in real-time and ismatched to the user's expected reading position in the electronic text.This enables playback effects to be configured for audio regions withoutknowledge of the end user reading speed.

The foregoing description of the invention includes preferred formsthereof. Modifications may be made thereto without departing from thescope of the invention as defined by the accompanying claims.

1-7. (canceled)
 8. A method for displaying electronic text andsynchronizing the playback of a soundtrack for the electronic text, thesoundtrack comprising multiple audio regions configured for playbackduring corresponding text regions of the electronic text, each audioregion being defined by audio data comprising: an audio track forplayback in the audio region, a start text position indicative of theword in the electronic text corresponding to where the playback of theaudio track is to begin; and a stop text position indicative of the wordin the electronic text corresponding to where playback of the audiotrack is to cease, and at least some audio regions further each beingdefined by pre-defined playback effect data that defines one or morecontrollable audio playback effect settings that are to be varied acrossthe audio region or a portion of the audio region based on the user'sreal-time text position, comprising: displaying at least a portion ofthe electronic text on a display; maintaining a reading position countercorresponding to the estimated text position indicative of the word inthe electronic text that the user is currently reading, the counterincrementing at a rate based on a configurable user reading speedvariable representing the user's reading speed; controlling playback ofthe audio regions of the soundtrack over an audio output system based onthe audio data and reading position counter to synchronise playback ofthe audio regions with the user's estimated reading position; andcontrolling audio playback effect settings of the audio output system inreal-time based on the reading position counter for any audio regionshaving pre-defined playback effect data such that the audio playbackeffect settings are varied in real-time based on the user's real-timetext position to match the pre-defined playback effect data.
 9. A methodaccording to claim 8 wherein the audio output system comprises aplurality of channels such that any overlapping audio regions canplayback concurrently on separate channels, and wherein the methodcomprises controlling independently the playback effect settings of eachchannel based on any pre-defined audio playback effect data associatedwith the overlapping audio regions.
 10. A method according to claim 8wherein the pre-configured playback effect data is defined by aprogrammable function the output of which represents a playback effectsetting that is dependent on a variable corresponding to the readingposition counter or a counter that is a product or function of thereading position counter.
 11. A method according to claim 8 wherein thepre-configured playback effect data associated with an audio region isin the form of a look-up table which defines a discrete playback effectsetting for a series of discrete text positions extending across atleast a portion of the audio region.
 12. A method according to claim 8wherein the pre-configured playback effect data associated with an audioregion is represented in the form of one or more envelopes that eachdefine a playback effect setting that is variable across at least aportion of the audio region, and where the shape of each envelope acrossthe audio region or portion of the audio region defines how itsassociated playback effect setting is varied across the audio region orportion of the audio region.
 13. A method according to claim 12 whereinthe pre-configured playback effect data comprises a volume envelope thatdefines a variable volume setting to be applied during playback of theaudio region, and controlling audio playback effect settings of theaudio output system comprises updating a volume setting of the audiooutput system in real-time to match the volume envelope based on theuser's real-time text position within the audio region.
 14. A methodaccording to claim 12 wherein the pre-configured playback effect datacomprises a panning envelope that defines a variable balance setting tobe applied during playback of the audio region, and controlling theaudio playback effect settings of the audio output system comprisesupdating a balance setting of the audio output system in real-time tomatch the panning envelope based on the user's real-time text positionwithin the audio region.
 15. A method according to claim 8 wherein thepre-configured playback effect data defines any one or more of thefollowing controllable audio playback effect settings: volume, balance,equalization, reverberation, distortion, and phase-shifting.
 16. \ Asystem for displaying electronic text and controlling playback of asoundtrack synchronized to the reading speed of a user comprising: adisplay configured to display the electronic text; a user interfaceconfigured to receive user input to control the system; an audio outputsystem that is operable to generate audio output for the user to hearand which has controllable audio playback effect settings for applyingaudio playback effects to the audio output; an electronic text displaymodule configured to process electronic text data representing theelectronic text and control display of the electronic text on thedisplay in response to user input from the user interface; a readingposition module configured to generate a reading position counterrepresenting the estimated current text position indicative of the wordin the electronic text that the user is currently reading, the readingposition counter incrementing at a rate based at least partly on aconfigurable user reading speed variable representing the user's readingspeed; and a soundtrack playback module configured to process soundtrackdata representing the soundtrack to control playback of the soundtrackover the audio output system, the soundtrack data defining multipleaudio regions for the soundtrack, each audio region being defined byaudio data comprising: an audio track for playback in the audio region;a start text position indicative of the word in the electronic textcorresponding to where the playback of the audio region is to begin; anda stop text position indicative of the word in the electronic textcorresponding to where the playback of the audio region is to cease; andat least some of the audio regions further each being defined bypre-defined playback effect data that defines one or more controllableaudio playback effect settings that are to be varied across an audioregion or a portion of the audio region based on the user's real-timetext position, and wherein the soundtrack playback module is configuredto coordinate playback of the audio regions based on the readingposition counter to synchronize playback of the audio regions of thesoundtrack with the user's reading of the electronic text, and thesoundtrack playback module being further configured to control the audioplayback effect settings of the audio output system in real-time basedon the reading position counter for any audio regions having associatedpre-defined playback effect data such that the audio playback effectsettings for each such audio region are varied in real-time based on theuser's real-time text position to match the pre-defined playback effectdata. 17-40. (canceled)
 41. A method according to claim 8 whereincontrolling audio playback effect settings of the audio output system inreal-time comprises varying the audio playback effect settings inreal-time at a frequency at least as high as the reading positioncounter rate.
 42. A method according to claim 8 wherein controllingaudio playback effect settings of the audio output system in real-timecomprises varying the audio playback effect settings to match thepredefined playback effect data at a frequency that is based on anupdate counter.
 43. A system according to claim 16 wherein the audiooutput system comprises a plurality of channels such that anyoverlapping audio regions can playback concurrently on separatechannels, and wherein the method comprises controlling independently theplayback effect settings of each channel based on any pre-defined audioplayback effect data associated with the overlapping audio regions. 44.A system according to claim 16 wherein the pre-configured playbackeffect data is defined by a programmable function the output of whichrepresents a playback effect setting that is dependent on a variablecorresponding to the reading position counter or a counter that is aproduct or function of the reading position counter.
 45. A systemaccording to claim 16 wherein the pre-configured playback effect dataassociated with an audio region is in the form of a look-up table whichdefines a discrete playback effect setting for a series of discrete textpositions extending across at least a portion of the audio region.
 46. Asystem according to claim 16 wherein the pre-configured playback effectdata associated with an audio region is represented in the form of oneor more envelopes that each define a playback effect setting that isvariable across at least a portion of the audio region, and where theshape of each envelope across the audio region or portion of the audioregion defines how its associated playback effect setting is variedacross the audio region or portion of the audio region.
 47. A systemaccording to claim 46 wherein the pre-configured playback effect datacomprises a volume envelope that defines a variable volume setting to beapplied during playback of the audio region, and controlling audioplayback effect settings of the audio output system comprises updating avolume setting of the audio output system in real-time to match thevolume envelope based on the user's real-time text position within theaudio region.
 48. A system according to claim 46 wherein thepre-configured playback effect data comprises a panning envelope thatdefines a variable balance setting to be applied during playback of theaudio region, and controlling the audio playback effect settings of theaudio output system comprises updating a balance setting of the audiooutput system in real-time to match the panning envelope based on theuser's real-time text position within the audio region.
 49. A systemaccording to claim 16 wherein the pre-configured playback effect datadefines any one or more of the following controllable audio playbackeffect settings: volume, balance, equalization, reverberation,distortion, and phase-shifting.
 50. A system according to claim 16wherein controlling audio playback effect settings of the audio outputsystem in real-time comprises varying the audio playback effect settingsin real-time at a frequency at least as high as the reading positioncounter rate.
 51. A system according to claim 16 wherein controllingaudio playback effect settings of the audio output system in real-timecomprises varying the audio playback effect settings to match thepredefined playback effect data at a frequency that is based on anupdate counter.